Pilot Testing and Preliminary Psychometric Validation of Situational Judgment Test of Resistance to Sunk Cost

Rahmat Hidayat; Muhammad Dwirifqi Kharisma Putra

doi:10.22146/jpsi.73737

Pilot Testing and Preliminary Psychometric Validation of Situational Judgment Test of Resistance to Sunk Cost

Rahmat Hidayat, Muhammad Dwirifqi Kharisma Putra

(Submitted 24 March 2022)

(Published 23 December 2022)

Abstract

The sunk-cost fallacy is an anomaly in decision-making that has been proven in various experimental studies. However, individual differences in the tendency to fall into sunk-cost fallacy have not been sufficiently reported. This preliminary study contains a psychometric evaluation report of the Resistance to Sunk Costs (RtSC) measurement instrument, a component of the Adult Decision Making Competence instrument (A-DMC; De Bruin, Parker, & Fischhoff, 2007) which is modified into Indonesian socio-cultural and economic contexts. The RtSC instrument uses a situational judgment test model, therefore it is adequate to measure the psychological construct of sunk-cost fallacy. The data analysis techniques used in this study were Item Factor Analysis (IFA) and graded response models. Respondents in this study were 217 students in Indonesia. The analysis results show that the 1-factor model of resistance to sunk costs fits the data. At the item level, based on the IFA, it was found that two out of ten items were found to be misfits. The results of the GRM analysis on the remaining eight items indicate that the items fit the GRM. However, the distinguishing power and reliability were found to be low. It can be concluded that this preliminary study provides a variety of important information as suggestions for improvement for our modified scale. Several theoretical and methodological implications are discussed.

Keywords

item response theory; situational judgment test; sunk cost; validation

Full Text: PDF

DOI: 10.22146/jpsi.73737

References

Arkes, H. R., & Ayton, P. (1999). The sunk cost and concorde effects: are humans less rational than lower animals? Psychological Bulletin, 125(5), 591-600. https://doi.org/10.1037/0033-2909.125.5.591

Arkes, H. R., & Blumer, C. (1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes, 35(1), 124-140. https://doi.org/10.1016/0749-5978(85)90049-4

Cai, L., & Monroe, S. (2014). A new statistic for evaluating item response theory models for ordinal data (CRESST Report 839). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). http://cresst.org/wp-content/uploads/R839.pdf Campion,

M. C., Ployhart, R. E., & MacKenzie, W. I. (2014). The state of research on situational judgment tests: a content analysis and directions for future research. Human Performance, 27(4), 283-310. https://doi.org/10.1080/08959285.2014.929693

Chalmers, R. P. (2012). mirt: a multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06

Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen's Q3: Identification of local dependence in the Rasch model using residual correlations. Applied Psychological Measurement, 41(3), 178-194.

Coleman, M. D. (2010). Sunk cost and commitment to medical treatment. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues, 29(2), 121-134. https://doi.org/10.1007/s12144-010-9077-7

Cook, K. F., Kallen, M. A., & Amtmann, D. (2009). Having a fit: impact of number of items and distribution of data on traditional criteria for assessing IRT’s unidimensionality assumption. Quality of Life Research, 18(4), 447–460. https://doi.org/10.1007/s11136-009-9464-4

Cunha, M., & Caldieraro, F. (2009). Sunk-cost effects on purely behavioral investments. Cognitive Science: A Multidisciplinary Journal, 33(1), 105-113. https://doi.org/10.1111/j.1551-6709.2008.01005.x

Davis, L. V., & Toseland, R. W. (1987). Group versus individual decision making. Social Work with Groups, 10(2), 95-110. https://doi.org/10.1300/J009v10n02_09

de Ayala, R. J. (2009). The theory and practice of item response theory. Guildford Press.

de Bruin, W. B., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult decision-making competence. Journal of Personality and Social Psychology, 9(5), 938-956. https://doi.org/10.1037/0022-3514.92.5.938

Dijkstra, K. A., & Hong, Y.-Y. (2019). The feeling of throwing good money after bad: the role of affective reaction in the sunk-cost fallacy. PLoS ONE, 14(1): e0209900. https://doi.org/10.1371/journal.pone.0209900

Fraley, C., Waller, N. G., & Brennan, K. A. (2000). An item response theory analysis of self-report measures of adult attachment. Journal of Personality and Social Psychology, 78(2), 350-365. https://doi.org/10.1037/0022-3514.78.2.350

Forero, C., & Maydeu-Olivares, A. (2009). Estimation of IRT graded response models: Limited versus full information methods. Psychological Methods, 14(3), 275–299. https://doi.org/10.1037/a0015825

Garland, H., & Newport, S. (1991). Effects of absolute and relative sunk costs on the decision to persist with a course of action. Organizational Behavior and Human Decision Processes, 48(1), 55-69. https://doi.org/10.1016/0749-5978(91)90005-E

Golubovich, J., Seybert, J., Martin-Raugh, M., Naemi, B., Vega, R. P., & Roberts, R. D. (2017). Assessing perceptions of interpersonal behavior with a video-based situational judgment test. International Journal of Testing, 17(3), 191-209. https://doi.org/10.1080/15305058.2016.1194275

Guenole, N., Chernyshenko, O. S., & Weekly, J. (2017). On designing construct driven situational judgement tests: some preliminary recommendations. International Journal of Testing, 17(3), 234-252. https://doi.org/10.1080/15305058.2017.1297817

Hafenbrack, A.C., Kinias, Z., & Barsade, S. G. (2014) Debiasing the mind through meditation: mindfulness and the sunk-cost bias. Psychological Science, 14(2), 369–376. https://doi.org/10.1177/0956797613503853

Haita-Falah, C. (2017). Sunk-cost fallacy and cognitive ability in individual decision-making. Journal of Economic Psychology, 58, 44-59. https://doi.org/10.1016/j.joep.2016.12.001 Heath, C. (1995). Escalation and de-escalation of commitment in response to sunk costs: the role of budgeting in mental accounting. Organizational behavior and human decision processes, 61(1), 38-54. https://doi.org/10.1006/obhd.1995.1029

Huggins-Manley, A. C. & Han, H. (2017). Assessing the sensitivity of weighted least squares model fit indexes to local dependence in item response theory models. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 331-340. https://doi.org/10.1080/10705511.2016.1247355

Jarmolowicz, D. P., Bickel, W. K., Sofis, M. J., Hatz, L. E., & Mueller, E. T. (2016). Sunk costs, psychological symptomology, and help seeking. SpringerPlus, 5: 1699. https://doi.org/10.1186/s40064-016-3402-z

Kamata, A., & Bauer, D. J. (2008). A note on the relation between factor analytic and item response theory models. Structural Equation Modeling: A Multidisciplinary Journal, 15(1), 136-153. https://doi.org/10.1080/10705510701758406

Kang, T., & Chen, T. T. (2008). Performance of the generalized S-X2 item fit index for polytomous IRT models. Journal of Educational Measurement, 45(4), 391-406. https://doi.org/10.1111/j.1745-3984.2008.00071.x

Keasey, K., & Moon, P. (2000). Sunk cost effects: a test of the importance of context. Economic Letters, 66(1), 55-58. https://doi.org/10.1016/S0165-1765(99)00179-2

Keefer, Q. A. W. (2017). The sunk-cost fallacy in the National Football League: salary cap value and playing time. Journal of Sport Economics, 18(3), 282-297. https://doi.org/10.1177/1527002515574515

Keefer, Q. A. W. (2019). Decision-maker beliefs and the sunk-cost fallacy: Major League Baseball’s final-offer salary arbitration and utilization. Journal of Economic Psychology, 75: 102080. https://doi.org/10.1016/j.joep.2018.06.002

Lievens, F. (2017). Construct-driven SJTs: toward an agenda for future research. International Journal of Testing, 17(3), 269-276. https://doi.org/10.1080/15305058.2017.1309857

Macaskill, A. C., & Hackenberg, T. D. (2012). The sunk effect with pigeons: Some determinants of decisions about persistence. Journal of the Experimental Analysis of Behavior, 97(1), 85–100. https://doi.org/10.1901/jeab.2012.97-85

Mair, P. (2018). Modern psychometrics with R. Springer International Publishing AG. https://doi.org/10.1007/978-3-319-93177-7

McCarthy, A. M., Schoorman, F. D., & Cooper, A. C. (1993). Reinvestment decisions by entrepreneurs: rational decision-making or escalation of commitment? Journal of Business Venturing, 8(1), 9-24.

McDaniel, M. A., & Nguyen, N. T. (2001). Situational judgement tests: a review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.

Meyers, E.A., Walker, A.C., Fugelsang, J.A., & Koehler, D.J. (2020). Reducing the number of nonnaïve participants in mechanical turk samples. Methods in Psychology, Advanced Online Publication. https://doi.org/10.1016/j.metip.2020.100032.

Moon, H. (2001). Looking forward and looking back: integrating completion and sunk-cost effects within an escalation-of-commitment progress decision. Journal of Applied Psychology, 86(1), 104–113. https://doi.org/10.1037/0021-9010.86.1.104

Motowidlo, S. J., & Tippins, N. (1993). Further studies of the low-fidelity simulations in the form of a situational inventory. Journal of Occupational and Organizational Psychology, 66(4), 337-344. https://doi.org/10.1111/j.2044-8325.1993.tb00543.x

Motowidlo, S. J., Lievens, F., & Ghosh, K. (2018). Prosocial implicit trait policies underlie performance on different situational judgment tests with interpersonal content. Human Performance, 31(4), 238-254. https://doi.org/10.1080/08959285.2018.1523909

Olaru, G., Burrus, J., MacCann, C., Zaromb, F. M., Wilhelm, O., & Roberts, R. D. (2019). Situational judgment tests as a method for measuring personality: development and validity evidence for a test of dependability. PLoS ONE, 14(2): e0211884. https://doi.org/10.1371/journal.pone.0211884

Oostrom, J. K., de Vries, R. E., & de Wit, M. (2019). Development and validation of a HEXACO situational judgement test. Human Performance, 32(1), 1-29. https://doi.org/10.1080/08959285.2018.1539856

Payne, J., & Wood, A. (2002). Individual decision making and group decision processes. The Journal of Psychology and Financial Markets, 3(2), 94-101. https://doi.org/10.1207/S15327760JPFM0302_04

Pilkonis, P. A., Kim, Y., Yu, L., & Morse, J. Q. (2014). Adult Attachment Ratings (AAR): An item response theory analysis. Journal of Personality Assessment, 96(4), 417-425. https://doi.org/10.1080/00223891.2013.832261

Rasch, G. (1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology, 19(1), 49–57. https://doi.org/10.1111/j.2044-8317.1966.tb00354.x

Rosenbaum, D. I., & Lamort, F. (1992). Entry, barriers, exit, and sunk costs: An analysis. Applied Economics, 24(3), 297-304. https://doi.org/10.1080/00036849200000142

Roth, S., Robbert, T., & Straus, L. (2014). On the sunk-cost effect in economic decision-making: a meta-analytic review. Business Research, 8, 99-138. https://doi.org/10.1007/s40685-014-0014-8

Rover, S., Wuerges, A. F. E., Tomazzia, E. C., & Borba, J. A. (2009). Sunk cost effect: does theoretical knowledge affects students’ decision processes? Brazilian Business Review, 6(3), 232-248.

Samejima, F. (1969). Estimation of ability using a response pattern of graded scores (Psychometrika Monograph No. 17). Psychometric Corporation. https://doi.org/10.1007/BF03372160

Schmitt, N., & Chan, D. (2006). Situational judgement tests: Method or construct? In J. A. Weekley, & R. E. Ployhart (Eds.), Situational judgement tests: theory, measurement and application (pp. 135-155). Lawrence Erlbaum Associates. Soman, D., &

Cheema, A. (2001). The effect of windfall gains on the sunk-cost effect. Marketing Letters, 12, 51-62. https://doi.org/10.1023/A:1008120003813

Stemler, S. E., & Sternberg, R. J. (2006). Using situational judgment tests to measure practical intelligence. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: theory, measurement, and application (p. 107–131). Lawrence Erlbaum Associates Publishers.

Strough, J., Mehta, C. M., McFall, J. P., & Schuller, K. L. (2008). Are older adults less subject to the sunk-cost fallacy than younger adults? Psychological Science, 19(7), 650–652. https://doi.org/10.1111/j.1467-9280.2008.02138.x

Thaler, R. (1980). Toward a positive theory of consumer choice. Journal of Economic Behavior & Organization, 1(1), 39-60. https://doi.org/10.1016/0167-2681(80)90051-7

Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211(4481), 453-458. https://doi.org/10.1126/science.7455683

Wang, J., & Keil, M. (2007). A meta-analysis comparing the sunk cost effect for IT and non-IT projects. Information Resources Management Journal, 20(3), 1-18.

Wang, J. ,& Wang, X. (2019). Structural equation modeling: applications using mplus (2nd ed.), John Wiley & Sons Ltd. https://doi.org/10.1002/9781119422730

Whitely, S. E., & Dawis, R. V. (1974). The nature objectivity with the Rasch model. Journal of Educational Measurement, 11(3), 163-178. https://doi.org/10.1111/j.1745-3984.1974.tb00988.x

Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: current approaches and future directions. Psychological Methods, 12(1), 58-79. https://doi.org/10.1037/1082-989X.12.1.58.

Yáñez, N., Bouzas, A., & Orduña, V. (2017). Rats behave optimally in a sunk cost task. Behavioural Processes, 140, 47–52. https://doi.org/10.1016/j.beproc.2017.04.003

Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125-145. https://doi.org/10.1177/014662168400800201

Zeelenberg, M., & van Dijk, E. (1997). A reverse sunk cost effect in risky decision making: Sometimes we have too much invested to gamble. Journal of Economic Psychology, 18(6), 677-691. https://doi.org/10.1016/S0167-4870(97)00029-9 Zeng,

J., Zhang, Q., Chen, C., Yu, R., & Gong, Q. (2013). An fMRI study on sunk cost effect. Brain Research, 1519, 63-70. https://doi.org/10.1016/j.brainres.2013.05.001

Zultan, R., Bar-Hillel, M., & Guy, N. (2010). When being wasteful appears better than feeling wasteful. Judgment and Decision Making, 5(7), 489–496.

Refbacks

There are currently no refbacks.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.