J Educ Eval Health Prof.  2018;15:26. 10.3352/jeehp.2018.15.26.

Linear programming method to construct equated item sets for the implementation of periodical computer-based testing for the Korean Medical Licensing Examination

• Seo DG
• Kim MG
• Kim NH
• Shin HS
• Kim HJ

• KMID: 2439282
• DOI: http://doi.org/10.3352/jeehp.2018.15.26

Abstract

PURPOSE
This study aimed to identify the best way of developing equivalent item sets and to propose a stable and effective management plan for periodical licensing examinations.
METHODS
Five pre-equated item sets were developed based on the predicted correct answer rate of each item using linear programming. These pre-equated item sets were compared to the ones that were developed with a random item selection method based on the actual correct answer rate (ACAR) and difficulty from item response theory (IRT). The results with and without common items were also compared in the same way. ACAR and the IRT difficulty were used to determine whether there was a significant difference between the pre-equating conditions.
RESULTS
There was a statistically significant difference in IRT difficulty among the results from different pre-equated conditions. The predicted correct answer rate was divided using 2 or 3 difficulty categories, and the ACAR and IRT difficulty parameters of the 5 item sets were equally constructed. Comparing the item set conditions with and without common items, including common items did not make a significant contribution to the equating of the 5 item sets.
CONCLUSION
This study suggested that the linear programming method is applicable to construct equated-item sets that reflect each content area. The suggested best method to construct equated item sets is to divide the predicted correct answer rate using 2 or 3 difficulty categories, regardless of common items. If pre-equated item sets are required to construct a test based on the actual data, several methods should be considered by simulation studies to determine which is optimal before administering a real test.

Keyword

Korea Medical Licensing Examination; Periodical examination; Linear programming; Predicted correct answer rate; Actual answer rate

MeSH Terms

Licensure*
Methods*
Programming, Linear*

Reference

References

1. Zhang J. Conditional covariance theory and detect for polytomous items. Psychometrika. 2007; 72:69–91. https://doi.org/10.1007/s11336-004-1257-7.

2. Robitzsch A. Robitzsch, A. sirt: Supplementary item response theory models: R package version 2.7-50 [Internet]. The R Foundation for Statistical Computing;2018. [cited 2018 May 30]. https://CRAN.R-project.org/package=sirt.

3. R Development Core Team. R: a language and environment for statistical computing [Internet]. Vienna: R Foundation for Statistical Computing;2008. [cited 2018 May 30]. Available from: http://www.R-project.org.

4. Fletcher RB. A review of linear programing and its application to the assessment tools for teaching and learning (asTTle) projects: technical report 5, project asTTle, University of Auckland, 2000 [Internet]. Berlin: Researchgate;2001. [cited 2018 May 10]. Available from:.

5. Rasch G. Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests [Internet]. Copenhagen: Danish Institute for Educational Research;1960. [cited 2018 May 1]. Available from: http://psycnet.apa.org/record/1962-07791-000.

6. Van der Linden WJ. Linear models for optimal test design. New York (NY): Springer;2005. https://doi.org/10.1007/0-387-29054-0.

7. Kingsbury GG, Zara AR. Procedures for selecting items for computerized adaptive tests. Appl Meas Educ. 1989; 2:359–375. https://doi.org/10.1207/s15324818ame0204_6.

8. Seo DG, Choi Y, Huh S. Usefulness of the DETECT program for assessing the internal structure of dimensionality in simulated data and results of the Korean nursing licensing examination. J Educ Eval Health Prof. 2017; 14:32. http://doi.org/10.3352/jeehp.2017.14.32.

9. Seo DG, Choi J. Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing Examination. J Educ Eval Health Prof. 2018; 15:14. http://doi.org/10.3352/jeehp.2018.15.14.

10. Kolen MJ, Brennan RL. Test equating, scaling, and linking: methods and practices. 3rd ed. New York (NY): Springer;2014. https://doi.org/10.1007/978-1-4939-0317-7.

11. Beguin AA, Hanson BA, Glas CAW. Effect of multidimensionality on separate and concurrent estimation in IRT equating. In : Proceedings of the Annual Meeting of the National Council on Measurement in Education 2000; 2000 Apr 25-27; New Orleans, USA. East Lansing (MI): National Council on Measurement in Education. 2000.

12. Kim SH, Cohen AS. A comparison of linking and concurrent calibration under item response theory. Appl Psychol Meas. 1998; 22:131–143. https://doi.org/10.1177/01466216980222003.

13. Seo DG. Overview and current management of computerized adaptive testing in licensing/certification examinations. J Educ Eval for Health Prof. 2017; 14:17. https://doi.org/10.3352/jeehp.2017.14.17.