J Educ Eval Health Prof.  2016;13:10. 10.3352/jeehp.2016.13.10.

Psychometric properties of a novel knowledge assessment tool of mechanical ventilation for emergency medicine residents in the northeastern United States

• Richards J
• Strout TD
• Seigel TA
• Wilcox SR

• KMID: 2413759
• DOI: http://doi.org/10.3352/jeehp.2016.13.10

Abstract

PURPOSE
Prior descriptions of the psychometric properties of validated knowledge assessment tools designed to determine Emergency medicine (EM) residents understanding of physiologic and clinical concepts related to mechanical ventilation are lacking. In this setting, we have performed this study to describe the psychometric and performance properties of a novel knowledge assessment tool that measures EM residents' knowledge of topics in mechanical ventilation.
METHODS
Results from a multicenter, prospective, survey study involving 219 EM residents from 8 academic hospitals in northeastern United States were analyzed to quantify reliability, item difficulty, and item discrimination of each of the 9 questions included in the knowledge assessment tool for 3 weeks, beginning in January 2013.
RESULTS
The response rate for residents completing the knowledge assessment tool was 68.6% (214 out of 312 EM residents). Reliability was assessed by both Cronbach's alpha coefficient (0.6293) and the Spearman-Brown coefficient (0.6437). Item difficulty ranged from 0.39 to 0.96, with a mean item difficulty of 0.75 for all 9 questions. Uncorrected item discrimination values ranged from 0.111 to 0.556. Corrected item-total correlations were determined by removing the question being assessed from analysis, resulting in a range of item discrimination from 0.139 to 0.498.
CONCLUSION
Reliability, item difficulty and item discrimination were within satisfactory ranges in this study, demonstrating acceptable psychometric properties of this knowledge assessment tool. This assessment indicates that this knowledge assessment tool is sufficiently rigorous for use in future research studies or for assessment of EM residents for evaluative purposes.

Keyword

Emergency medicine; Prospective studies; Psychometrics; Reproducibility of results; United States

MeSH Terms

Discrimination (Psychology)
Emergencies*
Emergency Medicine*
New England*
Prospective Studies
Psychometrics*
Reproducibility of Results
Respiration, Artificial*
United States

Reference

References

1. Wood S, Winters ME. Care of the intubated emergency department patient. J Emerg Med. 2011; 40:419–427. http://dx.doi.org/10.1016/j.jemermed.2010.02.021.

2. Fuller BM, Mohr NM, Miller CN, Deitchman AR, Levine BJ, Castagno N, Hassebroek EC, Dhedhi A, Scott-Wittenborn N, Grace E, Lehew C, Kollef MH. Mechanical ventilation and ARDS in the ED: a multi-center, observational, prospective, cross-sectional study. Chest. 2015; 148:365–374. http://dx.doi.org/10.1378/chest.14-2476.

3. Allison MG, Scott MC, Hu KM, Witting MD, Winters ME. High initial tidal volumes in emergency department patients at risk for acute respiratory distress syndrome. J Crit Care. 2015; 30:341–343. http://dx.doi.org/10.1016/j.jcrc.2014.12.004.

4. Brennan DF, Silvestri S, Sun JY, Papa L. Progression of emergency medicine resident productivity. Acad Emerg Med. 2007; 14:790–794. http://dx.doi.org/10.1111/j.1553-2712.2007.tb02353.x.

5. Cushman JT, Witting MD. Emergency medicine resident scheduling and patient exposure. Acad Emerg Med. 2003; 10:816–818. http://dx.doi.org/DOI:10.1197/aemj.10.7.816.

6. Gajic O, Frutos-Vivar F, Esteban A, Hubmayr RD, Anzueto A. Ventilator settings as a risk factor for acute respiratory distress syndrome in mechanically ventilated patients. Intensive Care Med. 2005; 31:922–926. http://dx.doi.org/10.1007/s00134-005-2625-1.

7. Hodder R, Lougheed MD, FitzGerald JM, Rowe BH, Kaplan AG, McIvor RA. Management of acute asthma in adults in the emergency department: assisted ventilation. CMAJ. 2010; 182:265–272. http://dx.doi.org/10.1503/cmaj.080073.

8. Warner KJ, Cuschieri J, Copass MK, Jurkovich GJ, Bulger EM. Emergency department ventilation effects outcome in severe traumatic brain injury. J Trauma. 2008; 64:341–347. http://dx.doi.org/10.1097/TA.0b013e318160dfb3.

9. Wilcox SR, Seigel TA, Strout TD, Schneider JI, Mitchell PM, Marcolini EG, Cocchi MN, Smithline HA, Lutfy-Clayton L, Mullen M, Ilgen JS, Richards JB. Emergency medicine residents’ knowledge of mechanical ventilation. J Emerg Med. 2015; 48:481–491. http://dx.doi.org/10.1016/j.jemermed.2014.09.059.

10. Goligher EC, Ferguson ND, Kenny LP. Core competency in mechanical ventilation: Development of educational objectives using the delphi technique. Crit Care Med. 2012; 40:2828–2832. http://dx.doi.org/10.1097/CCM.0b013e31825bc695.

11. Crocker L, Algina J. Introduction to classical and modern test theory. Mason, OH: Cengage Learning;2008.

12. Bland JM, Altman DG. Cronbach’s alpha. BMJ. 1997; 314:572. http://dx.doi.org/10.1136/bmj.314.7080.572.

13. Tavakol M, Dennick R. Psychometric evaluation of a knowledge based examination using Rasch analysis: an illustrative guide: AMEE guide no 72. Med Teach. 2013; 35:3838–3848. http://dx.doi.org/10.3109/0142159X.2012.737488.

14. Bechger TM, Maris G, Verstralen HHFM, Beguin AA. Using classical test theory in combination with item response theory. Appl Psychol Meas. 2003; 27:319–334. http://dx.doi.org/10.1177/0146621603257518.

15. Downing SM. Item response theory: applications of modern test theory in medical education. Med Educ. 2003; 37:739–745. http://dx.doi.org/10.1046/j.1365-2923.2003.01587.x.

16. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003; 37:830–837. http://dx.doi.org/10.1046/j.1365-2923.2003.01594.x.