J Educ Eval Health Prof.  2009;6:3.

Patient Simulation: A Literary Synthesis of Assessment Tools in Anesthesiology

• Edler AA
• Fanning RG
• Chen MI
• Claure R
• Almazan D
• Struyk B
• Seiden SC

Abstract

High-fidelity patient simulation (HFPS) has been hypothesized as a modality for assessing competency of knowledge and skill in patient simulation, but uniform methods for HFPS performance assessment (PA) have not yet been completely achieved. Anesthesiology as a field founded the HFPS discipline and also leads in its PA. This project reviews the types, quality, and designated purpose of HFPS PA tools in anesthesiology. We used the systematic review method and systematically reviewed anesthesiology literature referenced in PubMed to assess the quality and reliability of available PA tools in HFPS. Of 412 articles identified, 50 met our inclusion criteria. Seventy seven percent of studies have been published since 2000; more recent studies demonstrated higher quality. Investigators reported a variety of test construction and validation methods. The most commonly reported test construction methods included "modified Delphi Techniques" for item selection, reliability measurement using inter-rater agreement, and intra-class correlations between test items or subtests. Modern test theory, in particular generalizability theory, was used in nine (18%) of studies. Test score validity has been addressed in multiple investigations and shown a significant improvement in reporting accuracy. However the assessment of predicative has been low across the majority of studies. Usability and practicality of testing occasions and tools was only anecdotally reported. To more completely comply with the gold standards for PA design, both shared experience of experts and recognition of test construction standards, including reliability and validity measurements, instrument piloting, rater training, and explicit identification of the purpose and proposed use of the assessment tool, are required.

Keyword

High-Fidelity Patient Simulation; Anesthesiology; Patient Simulation; Performance Assessment; Systemic Review; Test Theory

MeSH Terms

Anesthesiology
Humans
Patient Simulation
Reproducibility of Results
Research Personnel

Figure

• 

  Fig. 1. Methods of item selection.

• 

  Fig. 2. Test refinement methods.

• 

  Fig. 3. Score reliability measures.

Reference

1. Thorndike EL. An introduction to the theory of mental and social measurement. New York: Science Press;1904.

2. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002; 287:226–35.

3. McClelland DC. Testing for competence rather than for “intelligence”. Am Psychol. 1973; 28:1–14.

4. Accreditation Council for Graduate Medical Education. Outcomes Project [Internet]. Chicago (IL): Accreditation Council for Graduate Medical Education;c2009. [cited 2009 Jan 20]. Available from: http://www.acgme.org/outcome/.

5. Tetzlaff JE. Assessment of competency in anesthesiology. Anesthesiology. 2007; 106:812–25.

6. Bland CJ, Meurer LN, Maldonado G. A systematic approach to conducting a non-statistical meta-analysis of research literature. Acad Med. 1995; 70:642–53.

7. Slavin RE. Perspectives on evidence-based research in education: What works? Issues in synthesizing educational program evaluations. Educ Res. 2008; 37:5–14.

8. Fritz PZ, Gray T, Flanagan B. Review of mannequin-based high-fidelity simulation in emergency medicine. Emerg Med Australas. 2008; 20:1–9.

9. Howard SK, Gaba DM, Fish KJ, Yang G, Sarnquist FH. Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med. 1992; 63:763–70.

10. Crocker LM, Algina J. Introduction to classical and modern test theory. 2nd ed. Victoria: Thomson Wadsworth;2006.

11. Berkenstadt H, Kantor GS, Yusim Y, et al. The feasibility of sharing simulation-based evaluation scenarios in anesthesiology. Anesth Analg. 2005; 101:1068–74.

12. Berkenstadt H, Ziv A, Gafni N, Sidi A. The validation process of incorporating simulation-based accreditation into the anesthesiology Israeli national board exams. Isr Med Assoc J. 2006; 8:728–33.

13. Blike GT, Christoffersen K, Cravero JP, Andeweg SK, Jensen J. A method for measuring system safety and latent errors associated with pediatric procedural sedation. Anesth Analg. 2005; 101:48–58.

14. Blum RH, Raemer DB, Carroll JS, Dufresne RL, Cooper JB. A method for measuring the effectiveness of simulation-based team training for improving communication skills. Anesth Analg. 2005; 100:1375–80.

15. Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability and validity of a simulation-based acute care skills assessment for medical students and residents. Anesthesiology. 2003; 99:1270–80.

16. Boulet JR, Marray D, Kras J, Woodhouse J. Setting performance standards for mannequin-based acute-care scenarios: an examineecentered approach. Simul Healthc. 2008; 3:72–81.

17. Byrne AJ, Jones JG. Responses to simulated anaesthetic emergencies by anaesthetists with different durations of clinical experience. Br J Anaesth. 1997; 78:553–6.

18. Byrne AJ, Sellen AJ, Jones JG. Errors on anaesthetic record charts as a measure of anaesthetic performance during simulated critical incidents. Br J Anaesth. 1998; 80:58–62.

19. Chopra V, Gesink BJ, de Jong J, Bovill JG, Spierdijk J, Brand R. Does training on an anaesthesia simulator lead to improvement in performance? Br J Anaesth. 1994; 73:293–7.

20. Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. Anesthesiology. 2001; 95:36–42.

21. Devitt JH, Kurrek MM, Cohen MM, et al. Testing the raters: interrater reliability of standardized anaesthesia simulator performance. Can J Anaesth. 1997; 44:924–8.

22. Devitt JH, Kurrek MM, Cohen MM, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg. 1998; 86:1160–4.

23. Farnsworth ST, Egan TD, Johnson SE, Westenskow D. Teaching sedation and analgesia with simulation. J Clin Monit Comput. 2000; 16:273–85.

24. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003; 90:580–8.

25. Forrest FC, Taylor MA, Postlethwaite K, Aspinall R. Use of a highfidelity simulator to develop testing of the technical performance of novice anaesthetists. Br J Anaesth. 2002; 88:338–44.

26. Gaba DM, DeAnda A. The response of anesthesia trainees to simulated critical incidents. Anesth Analg. 1989; 68:444–51.

27. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology. 1998; 89:8–18.

28. Harrison TK, Manser T, Howard SK, Gaba DM. Use of cognitive aids in a simulated anesthetic crisis. Anesth Analg. 2006; 103:551–6.

29. Howard SK, Gaba DM, Smith BE, et al. Simulation study of rested versus sleep-deprived anesthesiologists. Anesthesiology. 2003; 98:1345–55.

30. Hunt EA, Walker AR, Shaffner DH, Miller MR, Pronovost PJ. Simulation of in-hospital pediatric medical emergencies and cardiopulmonary arrests: highlighting the importance of the first 5 minutes. Pediatrics. 2008; 121:e34–43.

31. Jacobsen J, Lindekaer AL, Ostergaard HT, et al. Management of anaphylactic shock evaluated using a full-scale anaesthesia simulator. Acta Anaesthesiol Scand. 2001; 45:315–9.

32. Johnson KB, Syroid ND, Drews FA, et al. Part Task and variable priority training in first-year anesthesia resident education: a combined didactic and simulation-based approach to improve management of adverse airway and respiratory events. Anesthesiology. 2008; 108:831–40.

33. Lorraway PG, Savoldelli GL, Joo HS, Chandra DB, Chow R, Naik VN. Management of simulated oxygen supply failure: is there a gap in the curriculum? Anesth Analg. 2006; 102:865–7.

34. Morgan PJ, Cleave-Hogg D. Evaluation of medical students’ performances using the anesthesia simulator. Acad Med. 1999; 74:202.

35. Morgan PJ, Cleave-Hogg D. Evaluation of medical students’ performance using the anaesthesia simulator. Med Educ. 2000; 34:42–5.

36. Morgan PJ, Cleave-Hogg D, Desousa S, Lam-McCulloch J. Applying theory to practice in undergraduate education using high fidelity simulation. Med Teach. 2006; 28:e10–5.

37. Morgan PJ, Cleave-Hogg D, DeSousa S, Tarshis J. Identification of gaps in the achievement of undergraduate anesthesia educational objectives using high-fidelity patient simulation. Anesth Analg. 2003; 97:1690–4.

38. Morgan PJ, Cleave-Hogg D, DeSousa S, Tarshis J. High-fidelity patient simulation: validation of performance checklists. Br J Anaesth. 2004; 92:388–92.

39. Morgan PJ, Cleave-Hogg D, Guest CB. A comparison of global ratings and checklist scores from an undergraduate assessment using an anesthesia simulator. Acad Med. 2001; 76:1053–5.

40. Morgan PJ, Cleave-Hogg DM, Guest CB, Herold J. Validity and reliability of undergraduate performance assessments in an anesthesia simulator. Can J Anaesth. 2001; 48:225–33.

41. Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology. 2007; 106:907–15.

42. Murray D, Boulet J, Ziv A, Woodhouse J, Kras J, McAllister J. An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation. Med Educ. 2002; 36:833–41.

43. Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology. 2007; 107:705–13.

44. Murray DJ, Boulet JR, Kras JF, McAllister JD, Cox TE. A simulation-based acute skills performance assessment for anesthesia training. Anesth Analg. 2005; 101:1127–34.

45. Murray DJ, Boulet JR, Kras JF, Woodhouse JA, Cox T, McAllister JD. Acute care skills in anesthesia practice: a simulation-based resident performance assessment. Anesthesiology. 2004; 101:1084–95.

46. Olympio MA, Whelan R, Ford RP, Saunders IC. Failure of simulation training to change residents’ management of oesophageal intubation. Br J Anaesth. 2003; 91:312–8.

47. Ringsted C, Ostergaard D, Ravn L, Pedersen JA, Berlac PA, van der Vleuten CP. A feasibility study comparing checklists and global rating forms to assess resident performance in clinical skills. Med Teach. 2003; 25:654–8.

48. Rosenblatt MA, Abrams KJ. The use of a human patient simulator in the evaluation of and development of a remedial prescription for an anesthesiologist with lapsed medical skills. Anesth Analg. 2002; 94:149–53.

49. Savoldelli GL, Naik VN, Joo HS, et al. Evaluation of patient simulator performance as an adjunct to the oral examination for senior anesthesia residents. Anesthesiology. 2006; 104:475–81.

50. Scavone BM, Sproviero MT, McCarthy RJ, et al. Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology. 2006; 105:260–6.

51. Schwid HA, O’Donnell D. Anesthesiologists’ management of simulated critical incidents. Anesthesiology. 1992; 76:495–501.

52. Schwid HA, Rooke GA, Carline J, et al. Evaluation of anesthesia residents using mannequin-based simulation: a multiinstitutional study. Anesthesiology. 2002; 97:1434–44.

53. Schwid HA, Rooke GA, Ross BK, Sivarajan M. Use of a computerized advanced cardiac life support simulator improves retention of advanced cardiac life support guidelines better than a textbook review. Crit Care Med. 1999; 27:821–4.

54. Steadman RH, Coates WC, Huang YM, et al. Simulation-based training is superior to problem-based learning for the acquisition of critical assessment and management skills. Crit Care Med. 2006; 34:151–7.

55. Weller J, Merry A, Warman G, Robinson B. Anaesthetists’ management of oxygen pipeline failure: room for improvement. Anaesthesia. 2007; 62:122–6.

56. Weller JM, Bloch M, Young S, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. Br J Anaesth. 2003; 90:43–7.

57. Weller JM, Jolly B, Robinson B. Generalisability of behavioural skills in simulated anaesthetic emergencies. Anaesth Intensive Care. 2008; 36:185–9.

58. Weller JM, Robinson BJ, Jolly B, et al. Psychometric characteristics of simulation-based assessment in anaesthesia and accuracy of selfassessed scores. Anaesthesia. 2005; 60:245–50.

59. Wheeler DW, Degnan BA, Murray LJ, et al. Retention of drug administration skills after intensive teaching. Anaesthesia. 2008; 63:379–84.

60. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology. 2005; 103:241–8.

61. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull. 1955; 52:281–302.

62. Shavelson RJ, Webb NM. Generalizability theory: a primer. Newbury Park: Sage Publications Inc;1991.

63. Wigdor AK, Green BF. Performance assessment for the workplace. Washington (DC): National Academy Press;1991.

64. Hravnak M, Tuite P, Baldisseri M. Expanding acute care nurse practitioner and clinical nurse specialist education: invasive procedure training and human simulation in critical care. AACN Clin Issues. 2005; 16:89–104.

65. Kurrek MM, Devitt JH. The cost for construction and operation of a simulation centre. Can J Anaesth. 1997; 44:1191–5.

66. Lampotang S, Good ML, Westhorpe R, Hardcastle J, Carovano RG. Logistics of conducting a large number of individual sessions with a full-scale patient simulator at a scientific meeting. J Clin Monit. 1997; 13:399–407.

67. Seropian MA. General concepts in full scale simulation: getting started. Anesth Analg. 2003; 97:1695–705.

68. Tuoriniemi P, Schott-Baer D. Implementing a high-fidelity simulation program in a community college setting. Nurs Educ Perspect. 2008; 29:105–9.