Korean Circ J.  2020 Jan;50(1):72-84. 10.4070/kcj.2019.0105.

Development and External Validation of a Deep Learning Algorithm for Prognostication of Cardiovascular Outcomes

Affiliations
  • 1Division of Cardiology, Department of Internal Medicine, Ewha Womans University Seoul Hospital, Ewha Womans University College of Medicine, Seoul, Korea.
  • 2Ewha Womans University Graduate School, Seoul, Korea.
  • 3Division of Cardiology, Severance Cardiovascular Hospital, Yonsei University College of Medicine, Seoul, Korea. hjchang@yuhs.ac
  • 4Department of Preventive Medicine, Yonsei University College of Medicine, Seoul, Korea.
  • 5AI R&D Lab. of Selvas AI Inc., Seoul, Korea.
  • 6Department of Epidemiology, Erasmus MC, Rotterdam, The Netherlands.
  • 7School of Medicine, Faculty of Health, Universidad Industrial de Santander UIS, Bucaramanga, Colombia.
  • 8Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
  • 9Department of Radiology and Medicine, Weill Cornell Medical College, Dalio Institute of Cardiovascular Imaging, New York-Presbyterian Hospital, New York, NY, USA.
  • 10Severance Biomedical Science Institute, Yonsei University College of Medicine, Seoul, Korea.

Abstract

BACKGROUND AND OBJECTIVES
We aim to explore the additional discriminative accuracy of a deep learning (DL) algorithm using repeated-measures data for identifying people at high risk for cardiovascular disease (CVD), compared to Cox hazard regression.
METHODS
Two CVD prediction models were developed from National Health Insurance Service-Health Screening Cohort (NHIS-HEALS): a Cox regression model and a DL model. Performance of each model was assessed in the internal and 2 external validation cohorts in Koreans (National Health Insurance Service-National Sample Cohort; NHIS-NSC) and in Europeans (Rotterdam Study). A total of 412,030 adults in the NHIS-HEALS; 178,875 adults in the NHIS-NSC; and the 4,296 adults in Rotterdam Study were included.
RESULTS
Mean ages was 52 years (46% women) and there were 25,777 events (6.3%) in NHIS-HEALS during the follow-up. In internal validation, the DL approach demonstrated a C-statistic of 0.896 (95% confidence interval, 0.886-0.907) in men and 0.921 (0.908-0.934) in women and improved reclassification compared with Cox regression (net reclassification index [NRI], 24.8% in men, 29.0% in women). In external validation with NHIS-NSC, DL demonstrated a C-statistic of 0.868 (0.860-0.876) in men and 0.889 (0.876-0.898) in women, and improved reclassification compared with Cox regression (NRI, 24.9% in men, 26.2% in women). In external validation applied to the Rotterdam Study, DL demonstrated a C-statistic of 0.860 (0.824-0.897) in men and 0.867 (0.830-0.903) in women, and improved reclassification compared with Cox regression (NRI, 36.9% in men, 31.8% in women).
CONCLUSIONS
A DL algorithm exhibited greater discriminative accuracy than Cox model approaches. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT02931500

Keyword

Cardiovascular diseases; Artificial intelligence

MeSH Terms

Adult
Artificial Intelligence
Cardiovascular Diseases
Cohort Studies
Female
Follow-Up Studies
Humans
Insurance, Health
Learning*
Male
Mass Screening
National Health Programs

Figure

  • Figure 1 Formation of the development, internal validation, and external validation cohorts. CVD = cardiovascular disease; NHIS-HEALS = National Health Insurance Service-Health Screening Cohort; NHIS-NSC = National Health Insurance Service-National Sample Cohort. *Individuals with CVD and non-CVD were defined according to CVD occurrence during the mean 9.8 ± 2.2 years follow-up period to 2013; †123,601 out of 412,030 individuals were randomly selected as the final dataset to deal with the imbalanced data between CVD and non-CVD. During the matching process for the development and validation datasets, 288,429 non-CVD were excluded.

  • Figure 2 Predicted vs. observed probability of cardiovascular disease by deep learning in the internal validation and external validation cohorts.


Cited by  1 articles

Machine Learning: a New Opportunity for Risk Prediction
Osung Kwon, Wonjun Na, Young-Hak Kim
Korean Circ J. 2020;50(1):85-87.    doi: 10.4070/kcj.2019.0314.


Reference

1. Conroy RM, Pyörälä K, Fitzgerald AP, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003; 24:987–1003.
Article
2. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. BMJ. 2007; 335:136.
Article
3. D'Agostino RB Sr, Grundy S, Sullivan LM, Wilson P; CHD Risk Prediction Group. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA. 2001; 286:180–187.
4. Goldstein BA, Navar AM, Carter RE. Moving beyond regression techniques in cardiovascular risk prediction: applying machine learning to address analytic challenges. Eur Heart J. 2017; 38:1805–1814.
Article
5. Deo RC. Machine learning in medicine. Circulation. 2015; 132:1920–1930.
Article
6. Narain R, Saxena S, Goyal AK. Cardiovascular risk prediction: a comparative study of Framingham and quantum neural network based approach. Patient Prefer Adherence. 2016; 10:1259–1270.
Article
7. Khatibi V, Montazer GA. A fuzzy-evidential hybrid inference engine for coronary heart disease risk assessment. Expert Syst Appl. 2010; 37:8536–8542.
Article
8. Kukar M, Kononenko I, Grošelj C, Kralj K, Fettich J. Analysing and improving the diagnosis of ischaemic heart disease with machine learning. Artif Intell Med. 1999; 16:25–50.
Article
9. Seong SC, Kim YY, Park SK, et al. Cohort profile: the National Health Insurance Service-National Health Screening Cohort (NHIS-HEALS) in Korea. BMJ Open. 2017; 7:e016640.
Article
10. Seong SC, Kim YY, Khang YH, et al. Data resource profile: the National Health Information Database of the National Health Insurance Service in South Korea. Int J Epidemiol. 2017; 46:799–800.
11. Hofman A, Brusselle GG, Darwish Murad S, et al. The Rotterdam Study: 2016 objectives and design update. Eur J Epidemiol. 2015; 30:661–708.
Article
12. National Health Insurance Service. National Health Screening Statistical Yearbook 2014. Wonju: National Health Insurance Service;2015.
13. López V, Fernández A, García S, Palade V, Herrera F. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. Inf Sci. 2013; 250:113–141.
Article
14. Cho IJ, Sung JM, Chang HJ, Chung N, Kim HC. Incremental value of repeated risk factor measurements for cardiovascular disease prediction in middle-aged Korean adults: results from the NHIS-HEALS (National Health Insurance System-National Health Screening Cohort). Circ Cardiovasc Qual Outcomes. 2017; 10:004197.
Article
15. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997; 9:1735–1780.
Article
16. Liao L, Ahn HI. Combining deep learning and survival analysis for asset health management. Int J Progn Health Manag. 2016; 7:020.
Article
17. Lloyd-Jones DM, Leip EP, Larson MG, et al. Prediction of lifetime risk for cardiovascular disease by risk factor burden at 50 years of age. Circulation. 2006; 113:791–798.
Article
18. Pencina MJ, D'Agostino RB Sr, Larson MG, Massaro JM, Vasan RS. Predicting the 30-year risk of cardiovascular disease: the Framingham heart study. Circulation. 2009; 119:3078–3084.
19. Ramsay SE, Morris RW, Whincup PH, Papacosta AO, Thomas MC, Wannamethee SG. Prediction of coronary heart disease risk by Framingham and SCORE risk assessments varies by socioeconomic position: results from a study in British men. Eur J Cardiovasc Prev Rehabil. 2011; 18:186–193.
Article
20. Murphy TP, Dhangana R, Pencina MJ, D'Agostino RB Sr. Ankle-brachial index and cardiovascular risk prediction: an analysis of 11,594 individuals with 10-year follow-up. Atherosclerosis. 2012; 220:160–167.
Article
21. Paynter NP, Chasman DI, Buring JE, Shiffman D, Cook NR, Ridker PM. Cardiovascular disease risk prediction with and without knowledge of genetic variation at chromosome 9p21.3. Ann Intern Med. 2009; 150:65–72.
Article
22. Polonsky TS, McClelland RL, Jorgensen NW, et al. Coronary artery calcium score and risk classification for coronary heart disease prediction. JAMA. 2010; 303:1610–1616.
Article
23. Hinton G, Deng L, Yu D, et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag. 2012; 29:82–97.
Article
24. Kennedy EH, Wiitala WL, Hayward RA, Sussman JB. Improved cardiovascular risk prediction using nonparametric regression and electronic health record data. Med Care. 2013; 51:251–258.
Article
25. Waljee AK, Higgins PD. Machine learning in medicine: a primer for physicians. Am J Gastroenterol. 2010; 105:1224–1226.
Article
26. Jung K, Shah NH. Implications of non-stationarity on predictive modeling using EHRs. J Biomed Inform. 2015; 58:168–174.
Article
27. Ross EG, Shah NH, Dalman RL, Nead KT, Cooke JP, Leeper NJ. The use of machine learning for the identification of peripheral artery disease and future mortality risk. J Vasc Surg. 2016; 64:1515–1522.e3.
Article
28. Antman EM, Loscalzo J. Precision medicine in cardiology. Nat Rev Cardiol. 2016; 13:591–602.
Article
Full Text Links
  • KCJ
Actions
Cited
CITED
export Copy
Close
Share
  • Twitter
  • Facebook
Similar articles
Copyright © 2024 by Korean Association of Medical Journal Editors. All rights reserved.     E-mail: koreamed@kamje.or.kr