Healthc Inform Res.  2023 Oct;29(4):301-314. 10.4258/hir.2023.29.4.301.

Machine Learning for Benchmarking Critical Care Outcomes

• Atallah L
• Nabian M
• Brochini L
• Amelung PJ

• KMID: 2547683
• DOI: http://doi.org/10.4258/hir.2023.29.4.301

Abstract

Objectives
Enhancing critical care efficacy involves evaluating and improving system functioning. Benchmarking, a retrospective comparison of results against standards, aids risk-adjusted assessment and helps healthcare providers identify areas for improvement based on observed and predicted outcomes. The last two decades have seen the development of several models using machine learning (ML) for clinical outcome prediction. ML is a field of artificial intelligence focused on creating algorithms that enable computers to learn from and make predictions or decisions based on data. This narrative review centers on key discoveries and outcomes to aid clinicians and researchers in selecting the optimal methodology for critical care benchmarking using ML.
Methods
We used PubMed to search the literature from 2003 to 2023 regarding predictive models utilizing ML for mortality (592 articles), length of stay (143 articles), or mechanical ventilation (195 articles). We supplemented the PubMed search with Google Scholar, making sure relevant articles were included. Given the narrative style, papers in the cohort were manually curated for a comprehensive reader perspective.
Results
Our report presents comparative results for benchmarked outcomes and emphasizes advancements in feature types, preprocessing, model selection, and validation. It showcases instances where ML effectively tackled critical care outcome-prediction challenges, including nonlinear relationships, class imbalances, missing data, and documentation variability, leading to enhanced results.
Conclusions
Although ML has provided novel tools to improve the benchmarking of critical care outcomes, areas that require further research include class imbalance, fairness, improved calibration, generalizability, and long-term validation of published models.

Keyword

Benchmarking, Critical Care, Length of Stay, Machine Learning, Mortality, Ventilation

Reference

References

1. Rhodes A, Moreno RP, Azoulay E, Capuzzo M, Chiche JD, Eddleston J, et al. Prospectively defined indicators to improve the safety and quality of care for critically ill patients: a report from the Task Force on Safety and Quality of the European Society of Intensive Care Medicine (ESICM). Intensive Care Med. 2012; 38(4):598–605. https://doi.org/10.1007/s00134-011-2462-3.

2. Vincent JL, Marshall JC, Namendys-Silva SA, Francois B, Martin-Loeches I, Lipman J, et al. Assessment of the worldwide burden of critical illness: the intensive care over nations (ICON) audit. Lancet Respir Med. 2014; 2(5):380–6. https://doi.org/10.1016/S2213-2600(14)70061-X.

3. Higgins TL.Quantifying risk and benchmarking performance in the adult intensive care unit. J Intensive Care Med. 2007; 22(3):141–56. https://doi.org/10.1177/0885066607299520.

4. Braun JP, Kumpf O, Deja M, Brinkmann A, Marx G, Bloos F, et al. The German quality indicators in intensive care medicine 2013: second edition. Ger Med Sci. 2013. 11:Doc09. https://doi.org/10.3205/000177.

5. Kumpf O, Braun JP, Brinkmann A, Bause H, Bellgardt M, Bloos F, et al. Quality indicators in intensive care medicine for Germany: third edition 2017. Ger Med Sci. 2017. 15:Doc10. https://doi.org/10.3205/000251.

6. Brown SE, Ratcliffe SJ, Halpern SD.An empirical comparison of key statistical attributes among potential ICU quality indicators. Crit Care Med. 2014; 42(8):1821–31. https://doi.org/10.1097/CCM.0000000000000334.

7. Salluh JIF, Soares M, Keegan MT.Understanding intensive care unit benchmarking. Intensive Care Med. 2017; 43(11):1703–7. https://doi.org/10.1007/s00134-017-4760-x.

8. Higgins TL, Stark MM, Henson KN, Freeseman-Freeman L.Coronavirus disease 2019 ICU patients have higher-than-expected Acute Physiology and Chronic Health Evaluation-adjusted mortality and length of stay than viral pneumonia ICU patients. Crit Care Med. 2021; 49(7):e701–6. https://doi.org/10.1097/ccm.0000000000005012.

9. Verburg IW, Atashi A, Eslami S, Holman R, Abu-Hanna A, de Jonge E, et al. Which models can I use to predict adult ICU length of stay? A systematic review. Crit Care Med. 2017; 45(2):e222–31. https://doi.org/10.1097/ccm.0000000000002054.

10. van Sluisveld N, Bakhshi-Raiez F, de Keizer N, Holman R, Wester G, Wollersheim H, et al. Variation in rates of ICU readmissions and post-ICU in-hospital mortality and their association with ICU discharge practices. BMC Health Serv Res. 2017; 17(1):281. https://doi.org/10.1186/s12913-017-2234-z.

11. Seneff MG, Zimmerman JE, Knaus WA, Wagner DP, Draper EA. Predicting the duration of mechanical ventilation. The importance of disease and patient characteristics. Chest. 1996; 110(2):469–79. https://doi.org/10.1378/chest.110.2.469.

12. Shashikumar SP, Wardi G, Paul P, Carlile M, Brenner LN, Hibbert KA, et al. Development and prospective validation of a deep learning algorithm for predicting need for mechanical ventilation. Chest. 2021; 159(6):2264–73. https://doi.org/10.1016/j.chest.2020.12.009.

13. Malmgren J, Waldenstrom AC, Rylander C, Johannesson E, Lundin S.Long-term health-related quality of life and burden of disease after intensive care: development of a patient-reported outcome measure. Crit Care. 2021; 25(1):82. https://doi.org/10.1186/s13054-021-03496-7.

14. Higgins TL, Teres D, Nathanson B.Outcome prediction in critical care: the Mortality Probability Models. Curr Opin Crit Care. 2008; 14(5):498–505. https://doi.org/10.1097/mcc.0b013e3283101643.

15. Ferrando-Vivas P, Jones A, Rowan KM, Harrison DA.Development and validation of the new ICNARC model for prediction of acute hospital mortality in adult critical care. J Crit Care. 2017; 38:335–9. https://doi.org/10.1016/j.jcrc.2016.11.031.

16. Keuning BE, Kaufmann T, Wiersema R, Granholm A, Pettila V, Moller MH, et al. Mortality prediction models in the adult critically ill: a scoping review. Acta Anaesthesiol Scand. 2020; 64(4):424–42. https://doi.org/10.1111/aas.13527.

17. Strand K, Flaatten H.Severity scoring in the ICU: a review. Acta Anaesthesiol Scand. 2008; 52(4):467–78. https://doi.org/10.1111/j.1399-6576.2008.01586.x.

18. Siontis GC, Tzoulaki I, Ioannidis JP.Predicting death: an empirical evaluation of predictive tools for mortality. Arch Intern Med. 2011; 171(19):1721–6. https://doi.org/10.1001/archinternmed.2011.334.

19. Raj R, Luostarinen T, Pursiainen E, Posti JP, Takala RS, Bendel S, et al. Machine learning-based dynamic mortality prediction after traumatic brain injury. Sci Rep. 2019; 9(1):17672. https://doi.org/10.1038/s41598-019-53889-6.

20. Subudhi S, Verma A, Patel AB, Hardin CC, Khandekar MJ, Lee H, et al. Comparing machine learning algorithms for predicting ICU admission and mortality in COVID-19. NPJ Digit Med. 2021; 4(1):87. https://doi.org/10.1038/s41746-021-00456-x.

21. Silva I, Moody G, Scott DJ, Celi LA, Mark RG.Predicting in-hospital mortality of ICU patients: the PhysioNet/computing in cardiology challenge 2012. Comput Cardiol (2010). 2012; 39:245–8.

22. Barboi C, Tzavelis A, Muhammad LN.Comparison of severity of illness scores and artificial intelligence models that are predictive of intensive care unit mortality: meta-analysis and review of the literature. JMIR Med Inform. 2022; 10(5):e35293. https://doi.org/10.2196/35293.

23. Thorsen-Meyer HC, Nielsen AB, Nielsen AP, Kaas-Hansen BS, Toft P, Schierbeck J, et al. Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records. Lancet Digit Health. 2020; 2(4):e179–91. https://doi.org/10.1016/s2589-7500(20)30018-2.

24. Caicedo-Torres W, Gutierrez J.ISeeU: visually interpretable deep learning for mortality prediction inside the ICU. J Biomed Inform. 2019; 98:103269. https://doi.org/10.1016/j.jbi.2019.103269.

25. Aczon MD, Ledbetter DR, Laksana E, Ho LV, Wetzel RC.Continuous prediction of mortality in the PICU: a recurrent neural network model in a single-center dataset. Pediatr Crit Care Med. 2021; 22(6):519–29. https://doi.org/10.1097/pcc.0000000000002682.

26. Grnarova P, Schmidt F, Hyland SL, Eickhoff C. Neural document embeddings for intensive care patient mortality prediction [Internet]. Ithaca (NY): arXiv.org;2016. [cited at 2023 Sep 30]. Available from: http://arxiv.org/abs/1612.00467.

27. Ghassemi M, Pimentel MA, Naumann T, Brennan T, Clifton DA, Szolovits P, et al. A multivariate timeseries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data. Proc AAAI Conf Artif Intell. 2015; 2015:446–53.

28. Badawi O, Breslow MJ.Readmissions and death after ICU discharge: development and validation of two predictive models. PLoS One. 2012; 7(11):e48758. https://doi.org/10.1371/journal.pone.0048758.

29. Bhattacharya S, Rajan V, Shrivastava H.ICU mortality prediction: a classification algorithm for imbalanced datasets. Proc AAAI Conf Artif Intell. 2017; 31(1):1288–94. https://doi.org/10.1609/aaai.v31i1.10721.

30. Guo C, Liu M, Lu M.A dynamic ensemble learning algorithm based on K-means for ICU mortality prediction. Appl Soft Comput. 2021; 103:107166. https://doi.org/10.1016/j.asoc.2021.107166.

31. Kim S, Kim W, Park RW.A comparison of intensive care unit mortality prediction models through the use of data mining techniques. Healthc Inform Res. 2011; 17(4):232–43. https://doi.org/10.4258/hir.2011.17.4.232.

32. Awad A, Bader-El-Den M, McNicholas J, Briggs J.Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach. Int J Med Inform. 2017; 108:185–95. https://doi.org/10.1016/j.ijmedinf.2017.10.002.

33. Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ.Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med. 2015; 3(1):42–52. https://doi.org/10.1016/s2213-2600(14)70239-5.

34. Moser A, Reinikainen M, Jakob SM, Selander T, Pettila V, Kiiski O, et al. Mortality prediction in intensive care units including premorbid functional status improved performance and internal validity. J Clin Epidemiol. 2022; 142:230–41. https://doi.org/10.1016/j.jclinepi.2021.11.028.

35. El-Rashidy N, El-Sappagh S, Abuhmed T, Abdelrazek S, El-Bakry HM.Intensive care unit mortality prediction: an improved patient-specific stacking ensemble model. IEEE Access. 2020; 8:133541–64. https://doi.org/10.1109/ACCESS.2020.3010556.

36. Badawi O, Liu X, Hassan E, Amelung PJ, Swami S.Evaluation of ICU risk models adapted for use as continuous markers of severity of illness throughout the ICU stay. Crit Care Med. 2018; 46(3):361–7. https://doi.org/10.1097/ccm.0000000000002904.

37. Chiu CC, Wu CM, Chien TN, Kao LJ, Qiu JT.Predicting the mortality of ICU patients by topic model with machine-learning techniques. Healthcare (Basel). 2022; 10(6):1087. https://doi.org/10.3390/healthcare10061087.

38. Iwase S, Nakada TA, Shimada T, Oami T, Shimazui T, Takahashi N, et al. Prediction algorithm for ICU mortality and length of stay using machine learning. Sci Rep. 2022; 12(1):12912. https://doi.org/10.1038/s41598-022-17091-5.

39. Pang K, Li L, Ouyang W, Liu X, Tang Y.Establishment of ICU mortality risk prediction models with machine learning algorithm using MIMIC-IV database. Diagnostics (Basel). 2022; 12(5):1068. https://doi.org/10.3390/diagnostics12051068.

40. Safaei N, Safaei B, Seyedekrami S, Talafidaryani M, Masoud A, Wang S, et al. E-CatBoost: an efficient machine learning framework for predicting ICU mortality using the eICU Collaborative Research Database. PLoS One. 2022; 17(5):e0262895. https://doi.org/10.1371/journal.pone.0262895.

41. Stenwig E, Salvi G, Rossi PS, Skjaervold NK.Comparative analysis of explainable machine learning prediction models for hospital mortality. BMC Med Res Methodol. 2022; 22(1):53. https://doi.org/10.1186/s12874-022-01540-w.

42. Zhao S, Tang G, Liu P, Wang Q, Li G, Ding Z.Improving mortality risk prediction with routine clinical data: a practical machine learning model based on eICU patients. Int J Gen Med. 2023; 16:3151–61. https://doi.org/10.2147/ijgm.s391423.

43. Meiring C, Dixit A, Harris S, MacCallum NS, Brealey DA, Watkinson PJ, et al. Optimal intensive care outcome prediction over time using machine learning. PLoS One. 2018; 13(11):e0206862. https://doi.org/10.1371/journal.pone.0206862.

44. Davoodi R, Moradi MH.Mortality prediction in intensive care units (ICUs) using a deep rule-based fuzzy classifier. J Biomed Inform. 2018; 79:48–59. https://doi.org/10.1016/j.jbi.2018.02.008.

45. Marafino BJ, Park M, Davies JM, Thombley R, Luft HS, Sing DC, et al. Validation of prediction models for critical care outcomes using natural language processing of electronic health record data. JAMA Netw Open. 2018; 1(8):e185097. https://doi.org/10.1001/jamanet-workopen.2018.5097.

46. Purushotham S, Meng C, Che Z, Liu Y.Benchmarking deep learning models on large healthcare datasets. J Biomed Inform. 2018; 83:112–34. https://doi.org/10.1016/j.jbi.2018.04.007.

47. Shapley LS. A value for n-person games. Kuhn AW, Tucker HW, editors. Contributions to the theory of games. II. Princeton (NJ): Princeton University Press;1953. p. 307–18.

48. Li L, Liu G. In-hospital mortality prediction for ICU patients on large healthcare MIMIC datasets using class imbalance learning. In : Proceedings of 2020, 5th IEEE International Conference on Big Data Analytics (ICBDA); 2020 May 8–11; Xiamen, China. p. 90–3. https://doi.org/10.1109/ICBDA49040.2020.9101272.

49. Zimmerman JE, Kramer AA, McNair DS, Malila FM, Shaffer VL.Intensive care unit length of stay: benchmarking based on Acute Physiology and Chronic Health Evaluation (APACHE) IV. Crit Care Med. 2006; 34(10):2517–29. https://doi.org/10.1097/01.ccm.0000240233.01711.d9.

50. Vasilevskis EE, Kuzniewicz MW, Cason BA, Lane RK, Dean ML, Clay T, et al. Mortality probability model III and simplified acute physiology score II: assessing their value in predicting length of stay and comparison to APACHE IV. Chest. 2009; 136(1):89–101. https://doi.org/10.1378/chest.08-2591.

51. Peres IT, Hamacher S, Oliveira FLC, Bozza FA, Salluh JIF.Prediction of intensive care units length of stay: a concise review. Rev Bras Ter Intensiva. 2021; 33(2):183–7. https://doi.org/10.5935/0103-507x.20210025.

52. Bacchi S, Tan Y, Oakden-Rayner L, Jannes J, Kleinig T, Koblar S.Machine learning in the prediction of medical inpatient length of stay. Intern Med J. 2022; 52(2):176–85. https://doi.org/10.1111/imj.14962.

53. Harutyunyan H, Khachatrian H, Kale DC, Ver Steeg G, Galstyan A.Multitask learning and benchmarking with clinical time series data. Sci Data. 2019; 6(1):96. https://doi.org/10.1038/s41597-019-0103-9.

54. Houthooft R, Ruyssinck J, van der Herten J, Stijven S, Couckuyt I, Gadeyne B, et al. Predictive modelling of survival and length of stay in critically ill patients using sequential organ failure scores. Artif Intell Med. 2015; 63(3):191–207. https://doi.org/10.1016/j.artmed.2014.12.009.

55. Li C, Chen L, Feng J, Wu D, Zimeng W, Liu J, et al. Prediction of length of stay on the intensive care unit based on least absolute shrinkage and selection operator. IEEE Access. 2019; 7:110710–21. https://doi.org/10.1109/ACCESS.2019.2934166.

56. Sotoodeh M, Ho JC.Improving length of stay prediction using a hidden Markov model. AMIA Jt Summits Transl Sci Proc. 2019; 2019:425–34.

57. Gentimis T, Ala’J A, Durante A, Cook K, Steele R. Predicting hospital length of stay using neural networks on MIMIC III data. In : Proceedings of 2017 IEEE 15th International Conference on Dependable, Autonomic and Secure Computing, 15th International Conference on Pervasive Intelligence and Computing, 3rd International Conference on Big Data Intelligence and Computing and Cyber Science and Technology Congress (DASC/PiCom/DataCom/CyberSciTech); 2017 Nov 6–10; Orlando, FL. p. 1194–201. https://doi.org/10.1109/DASCPICom-DataCom-CyberSciTec.2017.191.

58. Ma X, Si Y, Wang Z, Wang Y.Length of stay prediction for ICU patients using individualized single classification algorithm. Comput Methods Programs Biomed. 2020; 186:105224. https://doi.org/10.1016/j.cmpb.2019.105224.

59. Muhlestein WE, Akagi DS, Davies JM, Chambless LB.Predicting inpatient length of stay after brain tumor surgery: developing machine learning ensembles to improve predictive performance. Neurosurgery. 2019; 85(3):384–93. https://doi.org/10.1093/neuros/nyy343.

60. Wu J, Lin Y, Li P, Hu Y, Zhang L, Kong G.Predicting prolonged length of ICU stay through machine learning. Diagnostics (Basel). 2021; 11(12):2242. https://doi.org/10.3390/diagnostics11122242.

61. Alghatani K, Ammar N, Rezgui A, Shaban-Nejad A.Predicting intensive care unit length of stay and mortality using patient vital signs: machine learning model development and validation. JMIR Med Inform. 2021; 9(5):e21347. https://doi.org/10.2196/21347.

62. Peres IT, Hamacher S, Cyrino Oliveira FL, Bozza FA, Salluh JI.Data-driven methodology to predict the ICU length of stay: a multicentre study of 99,492 admissions in 109 Brazilian units. Anaesth Crit Care Pain Med. 2022; 41(6):101142. https://doi.org/10.1016/j.accpm.2022.101142.

63. Weissman GE, Hubbard RA, Ungar LH, Harhay MO, Greene CS, Himes BE, et al. Inclusion of unstructured clinical text improves early prediction of death or prolonged ICU stay. Crit Care Med. 2018; 46(7):1125–32. https://doi.org/10.1097/ccm.0000000000003148.

64. Williford E, Haley V, McNutt LA, Lazariu V.Dealing with highly skewed hospital length of stay distributions: the use of Gamma mixture models to study delivery hospitalizations. PLoS One. 2020; 15(4):e0231825. https://doi.org/10.1371/journal.pone.0231825.

65. Peres IT, Hamacher S, Oliveira FLC, Thome AMT, Bozza FA.What factors predict length of stay in the intensive care unit? Systematic review and meta-analysis. J Crit Care. 2020; 60:183–94. https://doi.org/10.1016/j.jcrc.2020.08.003.

66. Sayed M, Riano D, Villar J.Predicting duration of mechanical ventilation in acute respiratory distress syndrome using supervised machine learning. J Clin Med. 2021; 10(17):3824. https://doi.org/10.3390/jcm10173824.

67. Kramer AA, Gershengorn HB, Wunsch H, Zimmerman JE.Variations in case-mix-adjusted duration of mechanical ventilation among ICUs. Crit Care Med. 2016; 44(6):1042–8. https://doi.org/10.1097/ccm.0000000000001636.

68. Kulkarni AR, Athavale AM, Sahni A, Sukhal S, Saini A, Itteera M, et al. Deep learning model to predict the need for mechanical ventilation using chest X-ray images in hospitalised patients with COVID-19. BMJ Innov. 2021; 7(2):261–70. https://doi.org/10.1136/bmjinnov-2020-000593.

69. Yu L, Halalau A, Dalal B, Abbas AE, Ivascu F, Amin M, et al. Machine learning methods to predict mechanical ventilation and mortality in patients with COVID-19. PLoS One. 2021; 16(4):e0249285. https://doi.org/10.1371/journal.pone.0249285.

70. Douville NJ, Douville CB, Mentz G, Mathis MR, Pancaro C, Tremper KK, et al. Clinically applicable approach for predicting mechanical ventilation in patients with COVID-19. Br J Anaesth. 2021; 126(3):578–89. https://doi.org/10.1016/j.bja.2020.11.034.

71. Karri R, Chen YP, Burrell AJC, Penny-Dimri JC, Broadley T, Trapani T, et al. Machine learning predicts the short-term requirement for invasive ventilation among Australian critically ill COVID-19 patients. Plops One. 2022; 17(10):e0276509. https://doi.org/10.1371/journal.pone.0276509.

72. Parreco J, Hidalgo A, Parks JJ, Kozol R, Rattan R.Using artificial intelligence to predict prolonged mechanical ventilation and tracheostomy placement. J Surg Res. 2018; 228:179–87. https://doi.org/10.1016/j.jss.2018.03.028.

73. Sauer CM, Dam TA, Celi LA, Faltys M, de la Hoz MAA, Adhikari L, et al. Systematic review and comparison of publicly available ICU data sets: a decision guide for clinicians and data scientists. Crit Care Med. 2022; 50(6):e581–8. https://doi.org/10.1097/ccm.0000000000005517.

74. Liu X, Badawi O.369: ICU length-of-stay models should account for the interaction between survival and patient severity. Crit Care Med. 2020; 48(1):166. https://doi.org/10.1097/01.ccm.0000619828.57026.19.

75. Leevy JL, Khoshgoftaar TM, Bauder RA, Seliya N.A survey on addressing high-class imbalance in big data. J Big Data. 2018; 5(1):42. https://doi.org/10.1186/s40537-018-0151-6.

76. Johnson JM, Khoshgoftaar TM.Survey on deep learning with class imbalance. J Big Data. 2019; 6(1):27. https://doi.org/10.1186/s40537-019-0192-5.

77. Zimmerman JE, Kramer AA, McNair DS, Malila FM.Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med. 2006; 34(5):1297–310. https://doi.org/10.1097/01.ccm.0000215112.84523.f0.

78. Roosli E, Bozkurt S, Hernandez-Boussard T.Peeking into a black box, the fairness and generalizability of a MIMIC-III benchmarking model. Sci Data. 2022; 9(1):24. https://doi.org/10.1038/s41597-021-01110-7.

79. Xu J, Glicksberg BS, Su C, Walker P, Bian J, Wang F.Federated learning for healthcare informatics. J Healthc Inform Res. 2021; 5(1):1–19. https://doi.org/10.1007/s41666-020-00082-4.

80. Lane-Fall MB, Neuman MD.Outcomes measures and risk adjustment. Int Anesthesiol Clin. 2013; 51(4):10–21. https://doi.org/10.1097/aia.0b013e3182a70a52.