J Educ Eval Health Prof.  2015;12:51. 10.3352/jeehp.2015.12.51.

Developing a situational judgment test blueprint for assessing the non-cognitive skills of applicants to the University of Utah School of Medicine, the United States

• Colbert-Getz JM
• Pippitt K
• Chan B

• KMID: 2402058
• DOI: http://doi.org/10.3352/jeehp.2015.12.51

Abstract

PURPOSE
The situational judgment test (SJT) shows promise for assessing the non-cognitive skills of medical school applicants, but has only been used in Europe. Since the admissions processes and education levels of applicants to medical school are different in the United States and in Europe, it is necessary to obtain validity evidence of the SJT based on a sample of United States applicants.
METHODS
Ninety SJT items were developed and Kane's validity framework was used to create a test blueprint. A total of 489 applicants selected for assessment/interview day at the University of Utah School of Medicine during the 2014-2015 admissions cycle completed one of five SJTs, which assessed professionalism, coping with pressure, communication, patient focus, and teamwork. Item difficulty, each item's discrimination index, internal consistency, and the categorization of items by two experts were used to create the test blueprint.
RESULTS
The majority of item scores were within an acceptable range of difficulty, as measured by the difficulty index (0.50-0.85) and had fair to good discrimination. However, internal consistency was low for each domain, and 63% of items appeared to assess multiple domains. The concordance of categorization between the two educational experts ranged from 24% to 76% across the five domains.
CONCLUSION
The results of this study will help medical school admissions departments determine how to begin constructing a SJT. Further testing with a more representative sample is needed to determine if the SJT is a useful assessment tool for measuring the non-cognitive skills of medical school applicants.

Keyword

Cognition; Communication; Judgment; School admission criteria; United States

MeSH Terms

Cognition
Discrimination (Psychology)
Education
Europe
Humans
Judgment*
School Admission Criteria
Schools, Medical
United States*
Utah*

Reference

1. Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007; 41:378–384. http://dx.doi.org/10.1111/j.1365-2929.2007.02709.x.

2. Association of American Medical Colleges. Data and research [Internet]. Washington (DC): U.S. Medical School Applications and Matriculants by School, State of Legal Residence, and Sex;2014. [cited 2015 Aug 22]. Available from: https://www.aamc.org/download/321442/data/factstable1.pdf.

3. Patterson F, Ashworth V, Zibarras L, Coan P, Kerrin M, O’Neill P. Evaluations of situational judgement tests to assess non-academic attributes in selection. Med Educ. 2012; 46:850–868. http://dx.doi.org/10.1111/j.1365-2923.2012.04336.x.

4. Lievens F, Sackett PR. The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. J Appl Psychol. 2012; 97:460–468. http://dx.doi.org/10.1037/a0025741.

5. Patterson F, Baron H, Carr V, Plint S, Lane P. Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Med Educ. 2009; 43:50–57. http://dx.doi.org/10.1111/j.1365-2923.2008.03238.x.

6. Mahesan N, Choudhury SM, Rymer J. Get ahead!: the situational judgement test. Boca Raton: Taylor & Frances Group;2012.

7. Metcalfe D, Dev H. Situational judgement test. 2nd ed. Oxford: Oxford University Press;2014.

8. Picard O, Allsopp G, Campbell L. Foundation programme: 250 SJTs for foundation year entry. London: ISC Medical;2012.

9. Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Med Educ. 2015; 49:560–575. http://dx.doi.org/10.1111/medu.12678.

10. Tavakol M, Dennick R. Post-examination analysis of objective tests. Med Teach. 2011; 33:447–458. http://dx.doi.org/10.3109/0142159X.2011.564682.

11. McDonald ME. The nurse educator’s guide to assessing learning outcomes. 3rd ed. Burlington (MA): Jones & Bartlett Publishers;2014.

12. Tavakol M, Dennick R. Post-examination interpretation of objective test data: monitoring and improving the quality of high-stakes examinations: AMEE Guide No. 66. Med Teach. 2012; 34:e161–e175. http://dx.doi/10.3109/0142159X.2012.651178.