Clin Orthop Surg.  2012 Jun;4(2):149-155. 10.4055/cios.2012.4.2.149.

Pitfalls and Important Issues in Testing Reliability Using Intraclass Correlation Coefficients in Orthopaedic Research

• Lee KM
• Lee J
• Chung CY
• Ahn S
• Sung KH
• Kim TW
• Lee HJ
• Park MS

• KMID: 1383877
• DOI: http://doi.org/10.4055/cios.2012.4.2.149

Abstract

BACKGROUND
Intra-class correlation coefficients (ICCs) provide a statistical means of testing the reliability. However, their interpretation is not well documented in the orthopedic field. The purpose of this study was to investigate the use of ICCs in the orthopedic literature and to demonstrate pitfalls regarding their use.
METHODS
First, orthopedic articles that used ICCs were retrieved from the Pubmed database, and journal demography, ICC models and concurrent statistics used were evaluated. Second, reliability test was performed on three common physical examinations in cerebral palsy, namely, the Thomas test, the Staheli test, and popliteal angle measurement. Thirty patients were assessed by three orthopedic surgeons to explore the statistical methods testing reliability. Third, the factors affecting the ICC values were examined by simulating the data sets based on the physical examination data where the ranges, slopes, and interobserver variability were modified.
RESULTS
Of the 92 orthopedic articles identified, 58 articles (63%) did not clarify the ICC model used, and only 5 articles (5%) described all models, types, and measures. In reliability testing, although the popliteal angle showed a larger mean absolute difference than the Thomas test and the Staheli test, the ICC of popliteal angle was higher, which was believed to be contrary to the context of measurement. In addition, the ICC values were affected by the model, type, and measures used. In simulated data sets, the ICC showed higher values when the range of data sets were larger, the slopes of the data sets were parallel, and the interobserver variability was smaller.
CONCLUSIONS
Care should be taken when interpreting the absolute ICC values, i.e., a higher ICC does not necessarily mean less variability because the ICC values can also be affected by various factors. The authors recommend that researchers clarify ICC models used and ICC values are interpreted in the context of measurement.

Keyword

Reliability; Intraclass correlation coefficient; Orthopaedic research

MeSH Terms

Adolescent
Biomedical Research/*methods/*standards
Cerebral Palsy
Child
Child, Preschool
Computer Simulation
Databases, Factual
Female
Humans
Male
Models, Theoretical
Orthopedics/*methods/*standards
Physical Examination
Range of Motion, Articular
Reproducibility of Results
Research Design
Statistics as Topic
Young Adult

Figure

• 

  Fig. 1 Intra-class correlation coefficient (ICC) is defined by the presented formula using between-target mean square (BMS), within-target mean square (WMS), and number of observers (k). BMS represents true subject variability, and WMS represents measurement error.

• 

  Fig. 2 The data sets were simulated to imitate physical examination situation. We intended to vary the ranges and variability of the data simultaneously. Intra-class correlation coefficient (ICC) is associated with between-group variation and within variation-group. The left lower panel (Data 7) was taken as the reference where means of observers were determined to be 45, 50, and 55 and we increased within-group variation horizontally and between-group variation vertically so that a total of nine data sets were generated based on a multivariate normal distribution. We increased within-group variation by inflating the diagonal term of a covariance matrix, which was shown in horizontal direction and consequently it resulted in increasing ranges. The off-diagonal terms were modified to affect between-group variation. While a slope of one observer is fixed, slopes of others were gradually increased compared to the reference slope and its trend was presented in vertical direction.

• 

  Fig. 3 Of the 143 orthopaedic articles using intra-class correlation coefficient (ICC), review articles, articles in other languages than English, and the articles not registered on the Journal Citation Report (JCR) index were excluded. Finally 92 articles were included.

Reference

1. Kelly MB. A review of the observational data-collection and reliability procedures reported in The Journal of Applied Behavior Analysis. J Appl Behav Anal. 1977. 10(1):97–101.

2. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977. 33(1):159–174.

3. Guyatt G, Rennie D. Users' guides to the medical literature: a manual for evidence-based clinical practice. 2002. Chicago: AMA Press.

4. Hunt RJ. Percent agreement, Pearson's correlation, and kappa as measures of inter-examiner reliability. J Dent Res. 1986. 65(2):128–130.

5. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979. 86(2):420–428.

6. Hays WL. Statistics for the social sciences. 1973. New York: Holt, Rinehart and Winston.

7. Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973. 33(3):613–619.

8. Gage JR, Schwartz MH, Koop SE, Novacheck TF. The identifiation and treatment of gait problems in cerebral palsy. 2009. 2nd ed. London: Mac Keith Press.

9. Thomas HO. Diseases of the hip, knee, and ankle joints, with their deformities, treated by a new and efficient method. 1876. Liverpool: Dobb.

10. Staheli LT. The prone hip extension test: a method of measuring hip flexion deformity. Clin Orthop Relat Res. 1977. (123):12–15.

11. American Education Research Association. American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing. 1985. Washington: American Psychological Association.

12. Lomnicki ZA. The standard error of Gini's mean difference. Ann Math Stat. 1952. 23(4):635–637.

13. Hendricks WA, Robey KW. The sampling distribution of the coefficient of variation. Ann Math Stat. 1936. 7(3):129–132.

14. R: a language and environment for statistical computing [Interent]. R Development Core Team. 2010. cited 2012 Mar 20. Vienna: R Foundation for Statistical Computing;Available from: http://www.R-project.org/.

15. Genz A, Bretz F, Miwa T, et al. mvtnorm: multivariate normal and t distributions [Internet]. 2012. cited 2012 Mar 20. Vienna: R Foundation;Available from: http://cran.r-project.org/web/packages/mvtnorm/index.html.

16. Revelle W. psych: procedures for psychological, psychometric, and personality research. c2011. cited 2012 Mar 20. Available from: http://personality-project.org/r/psych.manual.pdf.

17. Pappas N, Lawrence JT, Donegan D, Ganley T, Flynn JM. Intraobserver and interobserver agreement in the measurement of displaced humeral medial epicondyle fractures in children. J Bone Joint Surg Am. 2010. 92(2):322–327.

18. Lee KM, Chung CY, Kwon DG, Han HS, Choi IH, Park MS. Reliability of physical examination in the measurement of hip flexion contracture and correlation with gait parameters in cerebral palsy. J Bone Joint Surg Am. 2011. 93(2):150–158.