J Educ Eval Health Prof.  2017;14:12. 10.3352/jeehp.2017.14.12.

The impacts of computer adaptive testing from a variety of perspectives

• Kimura T

• KMID: 2406675
• DOI: http://doi.org/10.3352/jeehp.2017.14.12

Abstract

Computer adaptive testing (CAT) is a kind of tailored testing, in that it is a form of computer-based testing that is adaptive to each test-taker's ability level. In this review, the impacts of CAT are discussed from different perspectives in order to illustrate crucial points to keep in mind during the development and implementation of CAT. Test developers and psychometricians often emphasize the efficiency and accuracy of CAT in comparison to traditional linear tests. However, many test-takers report feeling discouraged after taking CATs, and this feeling can reduce learning self-efficacy and motivation. A trade-off must be made between the psychological experiences of test-takers and measurement efficiency. From the perspective of educators and subject matter experts, nonstatistical specifications, such as content coverage, content balance, and form length are major concerns. Thus, accreditation bodies may be faced with a discrepancy between the perspectives of psychometricians and those of subject matter experts. In order to improve test-takers' impressions of CAT, the author proposes increasing the target probability of answering correctly in the item selection algorithm even if doing so consequently decreases measurement efficiency. Two different methods, CAT with a shadow test approach and computerized multistage testing, have been developed in order to ensure the satisfaction of subject matter experts. In the shadow test approach, a full-length test is assembled that meets the constraints and provides maximum information at the current ability estimate, while computerized multistage testing gives subject matter experts an opportunity to review all test forms prior to administration.

Keyword

Computer adaptive testing; Measurement efficiency; Learning self-efficacy; Motivation; Content balance

MeSH Terms

Accreditation
Animals
Cats
Learning
Motivation
Retinoscopy

Reference

References

1. Lord F.M. Applications of item response theory to practical testing problems. Hillsdale (NJ): Lawrence Erlbaum;1980. p. 274.

2. Frick TW. Computerized adaptive mastery tests as expert systems. J Educ Comput Res. 1992; 8:187–213. https://doi.org/10.2190/j87v-6vwp-52g7-l4xx.

3. Rudner LM. An examination of decision-theory adaptive testing procedures. In : Proceedings of the annual meeting of the American Educational Research Association; 2002 Apr 1-5; New Orleans, USA. Washington (DC). American Educational Research Association. 2002.

4. Rudner LM, Guo F. Computer adaptive testing for small scale programs and instructional systems. J Appl Test Technol. 2011; 12:1–12.

5. Eignor DR. Education, tests and measures in. In : Kempf-Leonard K, editor. Encyclopedia of social measurement. Amsterdam: Elsevier;2005. p. 765–772.

6. Thissen D, Mislevy RJ. Testing algorithms. In : Wainer H, editor. Computerized adaptive testing: a primer. 2nd ed. Mahwah (NJ): Lawrence Erlbaum;2000. p. 101–133.

7. Weiss DJ, Kingsbury G. Application of computerized adaptive testing to educational problems. J Educ Meas. 1984; 21:361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x.

8. Thompson NA, Prometric T. A practitioner’s guide for variable-length computerized classification testing. Pract Assess Res Eval. 2007; 12:1–13.

9. Thompson NA, Weiss DJ. A framework for the development of computerized adaptive tests. Pract Assess Res Eval. 2011; 16:1–9.

10. Rudner LM. An on-line, interactive, computer adaptive testing tutorial [Internet]. [place unknown]: [publisher unknown];1998. [cited 2017 Apr 18]. Available from: http://echo.edres.org:8080/scripts/cat/catdemo.htm.

11. Kingsbury GG, Houser RL. Developing computerized adaptive tests for school children. In : Drasgow F, Olson-Buchanan JB, editors. Innovations in computerized assessment. Mahwah (NJ): Erlbaum;1998. p. 93–116.

12. Andrich D. Review of the book computerized adaptive testing: a primer. Psychometrika. 1995; 4:615–620.

13. Gershon RC. Test anxiety and item order: new concerns for item response theory. In : Wilson M, editor. Objective measurement: theory into practice. Norwood (NJ): Ablex;1992. p. 175–193.

14. Kimura T, Nagaoka K. Psychological aspects of CAT: how test-takers feel about CAT. In : Proceedings of the International Association for Computer Adaptive Testing Conference; 2011 Oct 3-5; Pacific Grove, USA. Woodbury, MN. International Association for Computer Adaptive Testing. 2011.

15. Kimura T, Nagaoka K. Psychological aspects of CAT: seeking item selection rules which do not decrease test takers’ learning self-efficacy and motivation. In : Proceedings of the International Association for Computer Adaptive Testing Conference; 2012 Aug 12-14; Sydney, Australia. Woodbury (MN). International Association for Computer Adaptive Testing. 2012.

16. Kimura T, Ohnishi A, Nagaoka K. Moodle UCAT: a computer-adaptive test module for Moodle based on the Rasch model. In : Proceedings of the fifth International Conference on Measurement in Health, Education, Psychology and Marketing; 2012 Jan 23-25; Perth, Australia.

17. Kimura T, Koyama Y. Implementation of small-scale in-house CAT with corpus-driven lexical item bank for ESP. In : Proceedings of the FLEAT VI 2015; 2015 Aug 11-15; Cambridge, USA.

18. Kimura T. Manipulation of target probability of computer adaptive tests. In : Proceedings of the Pacific Rim Objective Measurement Symposium 2016; 2016 Aug 1-3; Xi’an, China.

19. Mills CN. Development and introduction of a computer adaptive Graduate Record Examination General Test. In : Drasgow F, OlsonBuchanan JB, editors. Innovations in computerized assessment. Mahwah (NJ): Erlbaum;1999. p. 117–135.

20. Van der Linden WJ. Constrained Adaptive Testing with Shadow Tests. In : van der Linden WJ, Glas CAW, editors. Elements of adaptive testing. New York (NY): Springer;2010; 31–55.

21. Kingsbury GG, Zara AR. Procedures for selecting items for computerized adaptive tests. Appl Meas Educ. 1989; 2:359–375. https://doi.org/10.1207/s15324818ame0204_6.

22. Swanson L, Stocking ML. A model and heuristic for solving very large item selection problems. Appl Psychol Meas. 1993; 17:151–166. https://doi.org/10.1177/014662169301700205.

23. Glas CA, Wainer H, Bradlow ET. MML and EAP estimation in Testlet-Based Adaptive Testing. In : van der Linden WJ, Glas CA, editors. Computer adaptive testing: theory and practice. Dordrecht: Kluwer;2000. p. 271–288.

24. Vos HJ, Glas CA. Testlet-Based Adaptive Mastery Testing. In : van der Linden WJ, Glas CA, editors. Elements of adaptive testing. New York (NY): Springer;2010. p. 387–409.

25. Frey A, Seitz NN, Brandt S. Testlet-based multidimensional adaptive testing. Front Psychol. 2016; 7:1758. https://doi.org/10.3389/fpsyg.2016.01758.

26. Luecht RM. Implementing the computer-adaptive sequential testing (CAST) framework to mass produce high quality computer-adaptive and mastery tests. In : Proceedings of the annual meeting of the National Council on Measurement in Education; 2000 Apr 25-27; New Orleans, USA. East Lansing (MI). National Council on Measurement in Education. 2000.

27. Mead AD. An introduction to multistage testing. Appl Meas Educ. 2006; 19:185–187. https://doi.org/10.1207/s15324818ame1903_1.

28. Van der Linden WJ, Reese LM. A model for optimal constrained adaptive testing. Appl Psychol Meas. 1998; 22:259–270. https://doi.org/10.1177/01466216980223006.

29. Yan D, von Davier AA, Lewis C. Computerized multistage testing: theory and applications. Boca Raton (FL): CRC Press;2014. p. 504.

30. Melican GJ, Breithaupt K, Zhang Y. Designing and implementing a multistage adaptive test: the uniform CPA exam. In : van der Linden WJ, Glas CA, editors. Elements of adaptive testing. New York (NY): Springer;2010. p. 167–190.

31. Reckase RD. Multidimensional-item-response-theory. New York (NY): Springer;2009. p. 353.

32. Segall DO. Principles of multidimensional adaptive testing. In : van der Linden WJ, Glas CA, editors. Elements of adaptive testing. New York (NY): Springer;2010. p. 57–76.

33. Mulder J, van der Linden WJ. Multidimensional adaptive testing with Kullback-Leibler information item selection. In : van der Linden WJ, Glas CA, editors. Elements of adaptive testing. New York (NY): Springer;2010. p. 77–102.

34. McBride JR, Martin JT. Reliability and validity of adaptive ability tests in a military setting. In : Weiss DJ, editor. New horizons in testing. New York (NY): Academic Press;1983. p. 223–236.

35. Sympson JB, Hetter RD. Controlling item-exposure rates in computerized adaptive testing. In : Proceedings of the 27th annual meeting of the Military Testing Association; 1985 Oct 21-25; San Diego, USA.

36. Davey T, Parshall CG. New algorithms for item selection and exposure control with computerized adaptive testing. In : Proceedings of the annual meeting of the American Educational Research Association; 1995 Apr 18-22; San Francisco, USA. Washington (DC). American Educational Research Association. 1995.

37. Stocking ML, Lewis C. Controlling item exposure conditional on ability in computerized adaptive testing. J Educ Behav Stat. 1998; 23:57–75. https://doi.org/10.2307/1165348.

38. Chang SW, Ansley TN. A comparative study of item exposure control methods in computerized adaptive testing. J Educ Meas. 2003; 40:71–103. https://doi.org/10.1111/j.1745-3984.2003.tb01097.x.

39. Stocking ML, Lewis C. Methods of controlling the exposure of items in CAT. In : van der Linden WJ, Glas CA, editors. Computer adaptive testing: theory and practice. Dordrecht: Kluwer;2000. p. 163–182.

40. International Association of Computer Adaptive Testing. Bibliography [Internet]. Woodbury, MN: International Association for Computer Adaptive Testing;[cited 2017 Apr 18]. Available from: http://iacat.org/biblio.