Korean J Med Educ.  2007 Sep;19(3):185-195.

Item Analysis of Clinical Performance Examination Using Item Response Theory and Classical Test Theory

• Lim HS
• Lee YM
• Ahn DS
• Lee JY
• Im H

Abstract

PURPOSE: The objectives of this study were: 1) to analyze Clinical Performance Examination(CPX) items using item response theory(IRT) and classical test theory(CTT) and 2) to discuss how to apply and interpret these results in order to improve the quality of CPX items. In addition, we intended to explore statistical procedures in order to merge examination data from several different medical schools.
METHODS
The subject of the study was the 2005 CPX examination data from 10 medical schools located in Seoul and the Kyunggi province. For merging data from ten different medical schools, Levene's test for homogeneity of variances was used. Homogeneous group selection was conducted based on ANOVA or Kruskal-Wallis' test and Tukey's multiple comparisons appropriately. The generalized partial credit model was applied to analyze polytomous items and the 2-parameter logistic model was used to analyze dichotomous items.
RESULTS
Data from 8 medical schools were incorporated into the analysis. The result of the discrimination index by IRT was different from that of CTT in both polytomous and dichotomous items. Discrimination index from IRT tended to be lower than that of CTT. Difficulty index of dichotomous items of two models was correlated well with each other. However, for polytomous items, IRT model provided more information than CCT.
CONCLUSION
We discovered that the CPX items were mostly easy in terms of difficulty index, and the result from IRT and CCT model did not correlated well in the discrimination index. IRT may provide more detailed information for polytomous items, but the checklist and criteria of scoring system should be cautiously reviewed.

Keyword

Clinical performance examination; Item response theory; Classical test theory

MeSH Terms

Checklist
Discrimination (Psychology)
Gyeonggi-do
Logistic Models
Schools, Medical
Seoul