J Educ Eval Health Prof.  2018;15:32. 10.3352/jeehp.2018.15.32.

Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee

• Park J
• Ahn DS
• Yim MK
• Lee J

• KMID: 2439286
• DOI: http://doi.org/10.3352/jeehp.2018.15.32

Abstract

PURPOSE
This study aimed to compare the possible standard-setting methods for the Korean Radiological Technologist Licensing Examination, which has a fixed cut score, and to suggest the most appropriate method.
METHODS
Six radiological technology professors set standards for 250 items on the Korean Radiological Technologist Licensing Examination administered in December 2016 using the Angoff, Ebel, bookmark, and Hofstee methods.
RESULTS
With a maximum percentile score of 100, the cut score for the examination was 71.27 using the Angoff method, 62.2 using the Ebel method, 64.49 using the bookmark method, and 62 using the Hofstee method. Based on the Hofstee method, an acceptable cut score for the examination would be between 52.83 and 70, but the cut score was 71.27 using the Angoff method.
CONCLUSION
The above results suggest that the best standard-setting method to determine the cut score would be a panel discussion with the modified Angoff or Ebel method, with verification of the rated results by the Hofstee method. Since no standard-setting method has yet been adopted for the Korean Radiological Technologist Licensing Examination, this study will be able to provide practical guidance for introducing a standard-setting process.

Keyword

Licensure; Education; Radiological technologist; Standard setting; Modified-Angoff; Ebel; Bookmark; Hofstee

MeSH Terms

Education
Licensure*
Methods*
Technology, Radiologic

Figure

• 

  Fig. 1. Suggested standard-setting process.

• 

  Fig. 2. Range of the acceptable low and high cut scores.

Reference

References

1. Park HK. Study on setting a passing score on Korean National Medical Licensing Examination. Seoul: Korea Health Personnel Licensing Examination;2006. p. 130.

2. Yim M. Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination. Korean J Med Educ. 2018; 30:347–357. https://doi.org/10.3946/kjme.2018.110.

3. Cizek GJ, Bunch MB. Standard setting: a guide to establishing and evaluating performance standards on tests. Thousand Oaks (CA): Sage Publication Inc;2007. p. 368.

4. Angoff WH. Scales, norms, and equivalent scores. Princeton (NJ): Educational Testing Service;1984. p. 144.

5. Thorndike RL, Angoff WH. Educational measurement. 2nd ed. Washington (DC): American Council on Education;1971.

6. Ebel RL. Essential of educational measurement. Englewood Cliffs (NJ): Prentice Hall;1972; 622.

7. Mitzel HC, Lewis DM, Patz RL, Green DR. The bookmark procedure: psychological perspectives. In : Cizek GJ, editor. Setting performance standards: concepts, methods, and perspectives. Mahwah (NJ): Lawrence Erlbaum Associates Publishers;2001. p. 249–281.

8. Skaggs G, Tessema A. Item disordinality with the bookmark standard setting procedure. In : Proceedings of the Annual Meeting of the National Council on Measurement in Education; 2001 Apr 11-13; Seattle, USA. East Lansing (MI). National Council on Measurement in Education. 2001. 20.

9. Kim NJ. The standard setting of clinical performance examination (CPX) by modified Angoff, bookmark, and item-descriptor matching (IDM) method. Seoul: Ewha Womans University;2010. p. 278.

10. Hofstee WK. The case for compromise in educational selection and grading. In : Anderson SB, Helmick JS, editors. On educational testing. San Francisco (CA): Jossey-Bass;1983. p. 109–127.

11. Ferdous AA, Plake BS. Item selection strategy for reducing the number of items rated in an Angoff standard setting study. Educ Psychol Meas. 2007; 67:193–206. https://doi.org/10.1177/0013164406288160.

12. Ferdous AA, Plake BS. The use of subsets of test questions in an Angoff standard-setting method. Educ Psychol Meas. 2005; 65:185–201. https://doi.org/10.1177/0013164404264852.

13. Sireci SG, Patelis T, Rizavi S, Dillingham AM, Rodriguez G. Setting standards on a computerized-adaptive placement examination. In : Proceedings of the Annual Meeting of the National Council on Measurement in Education; 2000 Apr; New Orleans, USA. East Lansing (MI). National Council on Measurement in Education. 2000.

14. Lee YJ, Park JH, Cheong JW, Kim SJ, Kim YD. Study on standard setting methods applied to medical school class. J Educ Cult. 2017; 23:189–209.

15. Partchev I. Simple interface to the estimation and plotting of IRT models: R package version 0.1.6 [Internet]. Vienna: The R Foundation for Statistical Computing;2012. [cited 2018 Dec 13]. Available from: http://cran.r-project.org/web/packages/irtoys.

16. Psychometrics and Assessment Services. Technical report on the standard setting exercise for the medical council of Canada qualifying examination part II [Internet]. Ottawa (CA): Medical Council of Canada;2015. [cited 2018 Dec 13]. Available from: https://mcc.ca/media/MCCQE-Part-II_Standard-Setting-Report_July-2015.pdf.