J Educ Eval Health Prof.
2024;21(1):3. 10.3352/jeehp.2024.21.3.
Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study
- Frey-Vogel AS
1 - Dzara K2
- Gifford KA3
- Park YS4
- Berk J5
- Heinly A6
- Wolcott D7
- Hall DA1
- Elliott Scott-Vernaglia S1
- Sparger KA1
- Chung EYp6
- Affiliations
-
- 1Department of Pediatrics, Harvard Medical School and Massachusetts General Hospital, Boston, MA, USA
- 2Center for Educator Development, Advancement, and Research and Department of Family and Community Medicine, Saint Louis University School of Medicine, Saint Louis, MO, USA
- 3Department of Pediatrics, Cleveland Clinic Lerner College of Medicine at Case Western University, Cleveland, OH, USA
- 4Department of Medical Education, University of Illinois College of Medicine, Chicago, IL, USA
- 5Department of Pediatrics and Internal Medicine, The Warren Alpert Medical School of Brown University and Hasbro Children’s Hospital, Providence, RI, USA
- 6Department of Pediatrics, The Warren Alpert Medical School of Brown University and Hasbro Children’s Hospital, Providence, RI, USA
- 7Department of Pediatrics, Geisel School of Medicine at Dartmouth and Dartmouth-Hitchcock Medical Center, Lebanon, NH, USA
- KMID: 2553512
- DOI: http://doi.org/10.3352/jeehp.2024.21.3
Abstract
- Purpose
Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure.
Methods
Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables).
Results
Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019).
Conclusion
Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.
- Full Text Links
-
- Actions
-
Cited
- CITED
-
- Close
- Share
-
- Similar articles
-
- Development of the Clinical Teaching Effectiveness Questionnaire in the United States
- The applicability of a validated team-based learning student assessment instrument to assess United Kingdom pharmacy students' attitude toward team-based learning
- Development and psychometric evaluation of a 360-degree evaluation instrument to assess medical students’ performance in clinical settings at the emergency medicine department in Iran: a methodological study
- Validity of Resident Assessment Instrument-Minimum Data Set Home Care Version in Korea
- Validity of Instrument Development Research in Korean Nursing Research
