J Educ Eval Health Prof.  2018;15:20. 10.3352/jeehp.2018.15.20.

Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece

• Han KC

• KMID: 2439276
• DOI: http://doi.org/10.3352/jeehp.2018.15.20

Abstract

Computerized adaptive testing (CAT) technology is widely used in a variety of licensing and certification examinations administered to health professionals in the United States. Many more countries worldwide are expected to adopt CAT for their national licensing examinations for health professionals due to its reduced test time and more accurate estimation of a test-taker's performance ability. Continuous improvements to CAT algorithms promote the stability and reliability of the results of such examinations. For this reason, conducting simulation studies is a critically important component of evaluating the design of CAT programs and their implementation. This report introduces the principles of SimulCAT, a software program developed for conducting CAT simulation studies. The key evaluation criteria for CAT simulation studies are explained and some guidelines are offered for practitioners and test developers. A step-by-step tutorial example of a SimulCAT run is also presented. The SimulCAT program supports most of the methods used for the 3 key components of item selection in CAT: the item selection criterion, item exposure control, and content balancing. Methods for determining the test length (fixed or variable) and score estimation algorithms are also covered. The simulation studies presented include output files for the response string, item use, standard error of estimation, Newton-Raphson iteration information, theta estimation, the full response matrix, and the true standard error of estimation. In CAT simulations, one condition cannot be generalized to another; therefore, it is recommended that practitioners perform CAT simulation studies in each stage of CAT development.

Keyword

Adaptive Testing; Simulation; Computer software

MeSH Terms

Animals
Cats
Certification
Health Occupations
Licensure
United States

Figure

• 

  Fig. 1. SimulCAT GUI step 1: examinee/item data. Users can generate examinee and item data from normal, uniform, and beta distributions. SimulCAT also supports charts of the generated examinee and item data (the “Histogram” button and “Plot Item(s)” button). GUI, graphical user interface.

• 

  Fig. 2. SimulCAT GUI step 2: item selection. The software prompts users to choose item selection criteria, item exposure control, test length, and content balancing. GUI, graphical user interface.

• 

  Fig. 3. SimulCAT GUI step 3: test administration. Users specify the details of the testing environment of computerized adaptive testing, for example, the score estimation method, number of simultaneous test administrations, pretest item administration, and other research tools and output options. GUI, graphical user interface.

• 

  Fig. 4. SimulCAT output file 1: SimulCAT test administration data (*.sca). Each column shows the examinee’s ID, true theta value, test length, final theta estimate, the standard error of estimation, responses, and item ID, respectively.

• 

  Fig. 5. SimulCAT output file 2: SimulCAT item use data (*.scu). Each column shows the item ID, item exposure, and number of days after item retirement, respectively.

• 

  Fig. 6. Display of 5,000 simulees randomly drawn from a standard normal distribution (actual θ values differ each time), and 300 items generated for the item pool (item parameters differ each time, and are generated by clicking on the “Generate True Item Parameters” button).

• 

  Fig. 7. Input for step 2.

• 

  Fig. 8. Input for step 3. After filling in all necessary information, including the output file name, the user clicks on the “Run Simulation” button.

• 

  Fig. 9. SimulCAT output file with test administration information (*.sca). The values in columns 3 through 7 indicate the simulee ID, true θ, number of items administered, θ estimate, and standard error of estimation, respectively.

• 

  Fig. 10. SimulCAT output file with information about item exposure/usage. The first 2 columns indicate the item ID and number of item exposures, respectively.

• 

  Fig. 11. Conditional SEE (average SEE in each θ range). SEE, standard error of estimation.

• 

  Fig. 12. Conditional MAE (average MAE in each θ range). MAE, mean absolute error.

• 

  Fig. 13. Conditional bias (average bias within each θ range).

• 

  Fig. 14. Average test length in each θ range.

• 

  Fig. 15. Item exposure by a-value.

• 

  Fig. 16. Item exposure by b-value.

Reference

References

1. Han KCT. Components of the item selection algorithm in computerized adaptive testing. J Educ Eval Health Prof. 2018; 15:7. https://doi.org/10.3352/jeehp.2018.15.7.

2. Han KT. SimulCAT: Windows software for simulating computerized adaptive test administration. Appl Psychol Meas. 2012; 36:64–66. https://doi.org/10.1177/0146621611414407.

3. Han KT. Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests. Appl Psychol Meas. 2016; 40:289–301. https://doi.org/10.1177/0146621616631317.

4. Weiss DJ. Improving measurement quality and efficiency with adaptive testing. Appl Psychol Meas. 1982; 6:473–492. https://doi.org/10.1177/014662168200600408.

5. Chang HH, Ying Z. Alpha-stratified multistage computerized adaptive testing. Appl Psychol Meas. 1999; 23:211–222. https://doi.org/10.1177/01466219922031338.

6. Chang HH, Qian J, Ying Z. a-Stratified multistage computerized adaptive testing with b blocking. Appl Psychol Meas. 2001; 25:333–341. https://doi.org/10.1177/01466210122032181.

7. Chang HH, Ying Z. A global information approach to computerized adaptive testing. Appl Psychol Meas. 1996; 20:213–229. https://doi.org/10.1177/014662169602000303.

8. Veerkamp WJ, Berger MP. Some new item selection criteria for adaptive testing. J Educ Behav Stat. 1997; 22:203–226. https://doi.org/10.2307/1165378.

9. Han KT. A gradual maximum information ratio approach to item selection in computerized adaptive testing. Reston (VA): Graduate Management Admission Council;2009.

10. Han KT. An efficiency balanced information criterion for item selection in computerized adaptive testing. J Educ Meas. 2012; 49:225–246. https://doi.org/10.1111/j.1745-3984.2012.00173.x.

11. Kingsbury GG, Zara AR. Procedures for selecting items for computerized adaptive tests. Appl Meas Educ. 1989; 2:359–375. https://doi.org/10.1207/s15324818ame0204_6.

12. Sympson JB, Hetter RD. Controlling item-exposure rates in computerized adaptive testing. In : Proceedings of the 27th Annual Meeting of the Military Testing Association; 1985 Oct 21-25; San Diego, USA. San Diego (CA). Navy Personnel Research and Development Centre. 1985; 973–977.

13. Stocking ML, Lewis C. A new method of controlling item exposure in computerized adaptive testing. ETS Res Rep Ser. 1995; 1995:i–29. https://doi.org/10.1002/j.2333-8504.1995.tb01660.x.

14. Stocking ML, Lewis C. Controlling item exposure conditional on ability in computerized adaptive testing. J Educ Behav Stat. 1998; 23:57–75. https://doi.org/10.3102/10769986023001057.

15. Han KT. MSTGen: simulated data generator for multistage testing. Appl Psychol Meas. 2013; 37:666–668. https://doi.org/10.1177/0146621613499639.

16. Luecht RM, Nungester RJ. Some practical examples of computeradaptive sequential testing. J Educ Meas. 1998; 35:229–249. https://doi.org/10.1111/j.1745-3984.1998.tb00537.x.