Korean Circ J.  2019 Sep;49(9):856-863. 10.4070/kcj.2019.0015.

External Validation of 3 Risk Scores in Adults with Congenital Heart Disease

• Ramchandani BK
• Polo L
• Sánchez R
• Rey J
• González
• Díez J
• Aroca A

• KMID: 2455792
• DOI: http://doi.org/10.4070/kcj.2019.0015

Abstract

BACKGROUND AND OBJECTIVES
Adults with congenital heart disease (CHD) are an increasing group of patients thanks to the survival of over 85% of children with CHD. 20% of these patients shall warrant a surgical procedure during their life span. However, currently there is no one risk score that assess correctly the mortality of these procedures. Thus, we analyse the risk scores used at our institution.
METHODS
From May 1991 till June 2017, 608 procedures in adults with CHD were performed. The 3 risk scores (risk adjustment for congenital heart surgery [RACHS-1], Aristotle, and Euroscore I) of each procedure were analysed. We used area under the receiver operating characteristic curve (c-index) to measure model discrimination, and Hosmer-Lemeshow (H-L) statistic along with calibration plots to measure calibration.
RESULTS
There was no statistical difference between the area under the curve for the 3 scores (χ²=0.58 with 2 df, p=0.750). There was no evidence of lack of fit for RACHS-1 (H-L, χ²=2.61; p=0.271) and Aristotle score (H-L, χ²=5.69; p=0.459). However, there was evidence in lack of calibration in the Euroscore I scoring system (H-L, χ²=33.69; p<0.001). The calibration slope for RACHS-1 was 0.912, for Aristotle (stratified in risk groups) was −0.14 and for Euroscore 1 (stratified in risk groups) was 0.46.
CONCLUSIONS
RACHS-1 seems to be best risk scoring system for calculating mortality applied to surgery in adults with CHD.

Keyword

Congenital heart defect; Validation studies

MeSH Terms

Adult*
Calibration
Child
Discrimination (Psychology)
Heart Defects, Congenital*
Humans
Mortality
ROC Curve
Thoracic Surgery

Figure

• 

  Figure 1 Discrimination of the 3 risk score scales: ROC curves of the RACHS-1, Aristotle, and Euroscore I. All 3 risk scores showed a fair discrimination without any statistically significant differences. ROC = receiver operating characteristic; RACHS-1 = risk adjustment for congenital heart surgery.

• 

  Figure 2 Calibration plots of RACHS-1, Aristotle, and Euroscore I risk score scales. RACHS-1 seems to slightly underestimate the in-hospital mortality, while Aristotle slightly overestimates it. Euroscore-I shows an important miscalibration. RACHS-1 = risk adjustment for congenital heart surgery.

• 

  Figure 3 Calibration plots of Aristotle (1–5) and Euroscore I (low, medium, and high) stratified by risk categories. When the Aristotle and Euroscore-I are grouped in risk categories both show an important miscalibration.

Reference

1. Verheugt CL, Uiterwaal CS, Vaartjes I, et al. Chance of surgery in adult congenital heart disease. Eur J Prev Cardiol. 2017; 24:1319–1327.

2. Zomer AC, Verheugt CL, Vaartjes I, et al. Surgery in adults with congenital heart disease. Circulation. 2011; 124:2195–2201.

3. Kogon B, Oster M. Assessing surgical risk for adults with congenital heart disease: are pediatric scoring systems appropriate? J Thorac Cardiovasc Surg. 2014; 147:666–671.

4. Fuller S, Jacobs JP. Congenital heart surgery in adults: the challenge of estimation of risk of mortality. World J Pediatr Congenit Heart Surg. 2016; 7:436–439.

5. Hörer J, Kasnar-Samprec J, Cleuziou J, et al. Mortality following congenital heart surgery in adults can be predicted accurately by combining expert-based and evidence-based pediatric risk scores. World J Pediatr Congenit Heart Surg. 2016; 7:425–435.

6. Fuller SM, He X, Jacobs JP, et al. Estimating mortality risk for adult congenital heart surgery: an analysis of the society of thoracic surgeons congenital heart surgery database. Ann Thorac Surg. 2015; 100:1728–1735.

7. Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg. 2002; 123:110–118.

8. Lacour-Gayet F, Clarke D, Jacobs J, et al. The Aristotle score: a complexity-adjusted method to evaluate surgical results. Eur J Cardiothorac Surg. 2004; 25:911–924.

9. Roques F, Nashef SA, Michel P, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999; 15:816–822.

10. Jacobs ML, Jacobs JP, Jenkins KJ, Gauvreau K, Clarke DR, Lacour-gayet F. Stratification of complexity: the risk adjustment for congenital heart surgery-1 method and the Aristotle Complexity Score--past, present, and future. Cardiol Young. 2008; 18:163–168.

11. Jacobs JP, Jacobs ML, Maruszewski B, et al. Initial application in the EACTS and STS Congenital Heart Surgery Databases of an empirically derived methodology of complexity adjustment to evaluate surgical case mix and results. Eur J Cardiothorac Surg. 2012; 42:775–779.

12. Lafuente S, Trilla A, Bruni L, et al. Validation of the EuroSCORE probabilistic model in patients undergoing coronary bypass grafting. Rev Esp Cardiol. 2008; 61:589–594.

13. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014; 35:1925–1931.

14. Hickey GL, Blackstone EH. External model validation of binary clinical risk prediction models in cardiovascular and thoracic surgery. J Thorac Cardiovasc Surg. 2016; 152:351–355.

15. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009; 338:b605.

16. Nashef SA. Risk scores and how to evaluate them. Eur J Cardiothorac Surg. 2016; 50:519.

17. Kieser TM, Rose MS, Head SJ. Comparison of logistic EuroSCORE and EuroSCORE II in predicting operative mortality of 1125 total arterial operations. Eur J Cardiothorac Surg. 2016; 50:509–518.

18. Schafer JL. Multiple imputation: a primer. Stat Methods Med Res. 1999; 8:3–15.

19. Aroca A, Polo L, Pérez-Farinós N, et al. Risk factors for surgery of congenital heart disease in adults: twenty-two years of experience. Who should operate them? Arch Cardiol Mex. 2014; 84:262–272.

20. Aroca Á, Segovia J, Polo L, et al. Reintervenciones quirúrgicas y trasplante de órganos torácicos. Rev Esp Cardiol Supl. 2009; 9:98–107.