Healthc Inform Res.  2021 Jul;27(3):241-248. 10.4258/hir.2021.27.3.241.

Validation of an Automatic Tagging System for Identifying Respiratory and Hemodynamic Deterioration Events in the Intensive Care Unit

• Jeddah D
• Chen O
• Lipsky AM
• Forgacs A
• Celniker G
• Lilly CM
• Pessach IM

• KMID: 2519040
• DOI: http://doi.org/10.4258/hir.2021.27.3.241

Abstract

Objectives
Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers.
Methods
This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts’ eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system.
Results
The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers.
Conclusions
We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.

Keyword

Critical Care, Big Data, Respiratory Insufficiency, Clinical Deterioration, Artificial Intelligence

Figure

• 

  Figure 1 Process of patient selection and stratified over-sampling of critical events. To increase the prevalence of events of interest and improve statistical efficiency, stratified over-sampling of deterioration events was implemented. The dataset was divided into three subsets: patients with presumptive respiratory events, patients with presumptive hemodynamic events, and patients with neither of those events. To avoid duplications, each stay could only belong to one category; hence, stays with both a respiratory and a hemodynamic event were grouped according to the event that occurred earlier during the stay. Each set was then randomly sampled to yield 166–167 stays, stratified for basic demographic and clinical characteristics. In this manner, the 500-stay validation cohort included a higher proportion of patients with significant events than the general patient population, but with similar baseline demographics.

• 

  Figure 2 Distribution of events between stays. The distribution of events between stays as tagged by the expert reviewers or by the automatic tagging system. More than one intubation event was defined as occurring if an intubation event occurred at least 12 hours after a previous extubation event during that intensive care unit (ICU) stay. Similarly, a vasopressor initiation event was included if it was the first vasopressor initiation during a particular ICU stay, or if it was initiated at least 6 hours after ending previous vasopressor administration.

Reference

References

1. Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, et al. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med. 2016; 44(9):1639–48.

2. Ramachandran SK, Nafiu OO, Ghaferi A, Tremper KK, Shanks A, Kheterpal S. Independent predictors and outcomes of unanticipated early postoperative tracheal intubation after nonemergent, noncardiac surgery. Anesthesiology. 2011; 115(1):44–53.

3. Skrobik Y, Kavanagh BP. Scoring systems for the critically ill: use, misuse and abuse. Can J Anaesth. 2006; 53(3):432–6.

4. Vincent JL, Opal SM, Marshall JC. Ten reasons why we should NOT use severity scores as entry criteria for clinical trials or in our treatment decisions. Crit Care Med. 2010; 38(1):283–7.

5. McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007; (3):CD005529.

6. Berlot G, Pangher A, Petrucci L, Bussani R, Lucangelo U. Anticipating events of in-hospital cardiac arrest. Eur J Emerg Med. 2004; 11(1):24–8.

7. Blackwell JN, Keim-Malpass J, Clark MT, Kowalski RL, Najjar SN, Bourque JM, et al. Early detection of in-patient deterioration: one prediction model does not fit all. Crit Care Explor. 2020; 2(5):e0116.

8. Pollard TJ, Celi LA. Enabling machine learning in critical care. ICU Manag Pract. 2017; 17(3):198–9.

9. Johnson AE, Mark RG. Real-time mortality prediction in the Intensive Care Unit. AMIA Annu Symp Proc. 2018; 2017:994–1003.

10. Bailly S, Meyfroidt G, Timsit JF. What’s new in ICU in 2050: big data and machine learning. Intensive Care Med. 2018; 44(9):1524–7.

11. Zimlichman E, Szyper-Kravitz M, Shinar Z, Klap T, Levkovich S, Unterman A, et al. Early recognition of acutely deteriorating patients in non-intensive care units: assessment of an innovative monitoring technology. J Hosp Med. 2012; 7(8):628–33.

12. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010; 363(22):2124–34.

13. Chong JL, Low LL, Chan DY, Shen Y, Thin TN, Ong ME, et al. Can we understand population healthcare needs using electronic medical records? Singapore Med J. 2019; 60(9):446–53.

14. Hanskamp-Sebregts M, Zegers M, Vincent C, van Gurp PJ, de Vet HC, Wollersheim H. Measurement of patient safety: a systematic review of the reliability and validity of adverse event detection with record review. BMJ Open. 2016; 6(8):e011078.

15. Cao H, Lee KP, Ennett CM, Eshelman L, Nielsen L, Saeed M, et al. Heuristics to determine ventilation times of ICU patients from the MIMIC-II database. Annu Int Conf IEEE Eng Med Biol Soc. 2010; 2010:5213–6.

16. Jeffery AD, Dietrich MS, Fabbri D, Kennedy B, Novak LL, Coco J, et al. Advancing in-hospital clinical deterioration prediction Models. Am J Crit Care. 2018; 27(5):381–91.

17. Ueno R, Xu L, Uegami W, Matsui H, Okui J, Hayashi H, et al. Value of laboratory results in addition to vital signs in a machine learning algorithm to predict in-hospital cardiac arrest: a single-center retrospective cohort study. PLoS One. 2020; 15(7):e0235835.

18. Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, Sundermann SH, et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018; 6(12):905–14.

19. Thorsen-Meyer HC, Nielsen AB, Nielsen AP, Kaas-Hansen BS, Toft P, Schierbeck J, et al. Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit: a retrospective study of high-frequency data in electronic patient records. Lancet Digit Health. 2020; 2(4):e179–e191.

20. Bulgarelli L, Deliberato RO, Johnson AE. Prediction on critically ill patients: the role of “big data”. J Crit Care. 2020; 60:64–8.

21. Fleuren LM, Klausch TL, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020; 46(3):383–400.