J Prev Med Public Health.  2015 Sep;48(5):239-248. 10.3961/jpmph.14.049.

Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events

• Ock M
• Lee SI
• Jo MW
• Lee JY
• Kim SH

• KMID: 2381474
• DOI: http://doi.org/10.3961/jpmph.14.049

Abstract

OBJECTIVES
The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events.
METHODS
We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later.
RESULTS
In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89).
CONCLUSIONS
In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.

Keyword

Adverse event; Patient safety; Intra-rater reliability; Inter-rater reliability; Medical record review

MeSH Terms

Drug-Related Side Effects and Adverse Reactions/classification/*pathology
Hospitals, General
Humans
Medical Records
Nurses/psychology
Physicians/psychology
Retrospective Studies