Healthc Inform Res.  2018 Apr;24(2):139-147. 10.4258/hir.2018.24.2.139.

Using Statistical and Machine Learning Methods to Evaluate the Prognostic Accuracy of SIRS and qSOFA

• Gupta A
• Liu T
• Shepherd S
• Paiva W

• KMID: 2418182
• DOI: http://doi.org/10.4258/hir.2018.24.2.139

Abstract

OBJECTIVES
The objective of this study was to compare the performance of two popularly used early sepsis diagnostic criteria, systemic inflammatory response syndrome (SIRS) and quick Sepsis-related Organ Failure Assessment (qSOFA), using statistical and machine learning approaches.
METHODS
This retrospective study examined patient visits in Emergency Department (ED) with sepsis related diagnosis. The outcome was 28-day in-hospital mortality. Using odds ratio (OR) and modeling methods (decision tree [DT], multivariate logistic regression [LR], and naïve Bayes [NB]), the relationships between diagnostic criteria and mortality were examined.
RESULTS
Of 132,704 eligible patient visits, 14% died within 28 days of ED admission. The association of qSOFA ≥2 with mortality (OR = 3.06; 95% confidence interval [CI], 2.96-3.17) greater than the association of SIRS ≥2 with mortality (OR = 1.22; 95% CI, 1.18-1.26). The area under the ROC curve for qSOFA (AUROC = 0.70) was significantly greater than for SIRS (AUROC = 0.63). For qSOFA, the sensitivity and specificity were DT = 0.39, LR = 0.64, NB = 0.62 and DT = 0.89, LR = 0.63, NB = 0.66, respectively. For SIRS, the sensitivity and specificity were DT = 0.46, LR = 0.62, NB = 0.62 and DT = 0.70, LR = 0.59, NB = 0.58, respectively.
CONCLUSIONS
The evidences suggest that qSOFA is a better diagnostic criteria than SIRS. The low sensitivity of qSOFA can be improved by carefully selecting the threshold to translate the predicted probabilities into labels. These findings can guide healthcare providers in selecting risk-stratification measures for patients presenting to an ED with sepsis.

Keyword

Sepsis; Systemic Inflammatory Response Syndrome; Severity of Illness Index; Medical Informatics; Artificial Intelligence

MeSH Terms

Artificial Intelligence
Bays
Diagnosis
Emergency Service, Hospital
Health Personnel
Hospital Mortality
Humans
Logistic Models
Machine Learning*
Medical Informatics
Methods*
Mortality
Odds Ratio
Retrospective Studies
ROC Curve
Sensitivity and Specificity
Sepsis
Severity of Illness Index
Systemic Inflammatory Response Syndrome
Trees

Figure

• 

  Figure 1 Encounter flow chart.

• 

  Figure 2 Distribution of encounters across SIRS and qSOFA scores. SIRS: systemic inflammatory response syndrome, qSOFA: quick Sepsis-related Organ Failure Assessment.

• 

  Figure 3 Distribution of mortality across SIRS and qSOFA scores. SIRS: systemic inflammatory response syndrome, qSOFA: quick Sepsis-related Organ Failure Assessment.

• 

  Figure 4 Odds ratio for in-hospital mortality for each decile of baseline risk. SIRS: systemic inflammatory response syndrome, qSOFA: quick Sepsis-related Organ Failure Assessment.

• 

  Figure 5 The area under the receiver operating characteristic (AUROC) curve using multivariate logistic regression. SIRS: systemic inflammatory response syndrome, qSOFA: quick Sepsis-related Organ Failure Assessment.

Reference

1. Freund Y, Lemachatti N, Krastinova E, Van Laer M, Claessens YE, Avondo A, et al. Prognostic accuracy of Sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA. 2017; 317(3):301–308.

2. Fleischmann C, Thomas-Rueddel DO, Hartmann M, Hartog CS, Welte T, Heublein S, et al. Hospital incidence and mortality rates of sepsis: an analysis of hospital episode (DRG) statistics in Germany from 2007 to 2013. Dtsch Arztebl Int. 2016; 113(10):159–166.

3. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive conditions by payer, 2011 (Statistical Brief #160). Rockville (MD): Agency for Healthcare Research and Quality;2006.

4. Castellanos-Ortega A, Suberviola B, Garcia-Astudillo LA, Holanda MS, Ortiz F, Llorca J, et al. Impact of the Surviving Sepsis Campaign protocols on hospital length of stay and mortality in septic shock patients: results of a three-year follow-up quasi-experimental study. Crit Care Med. 2010; 38(4):1036–1043.

5. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016; 315(8):801–810.

6. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest. 1992; 101(6):1644–1655.

7. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Heidelberg, Germany: Springer;1996.

8. Gardner-Thorpe J, Love N, Wrightson J, Walsh S, Keeling N. The value of Modified Early Warning Score (MEWS) in surgical in-patients: a prospective observational study. Ann R Coll Surg Engl. 2006; 88(6):571–575.

9. Kim YM, Kathuria P, Delen D. Machine learning to compare frequent medical problems of African American and Caucasian diabetic kidney patients. Healthc Inform Res. 2017; 23(4):241–248.

10. Kuo CY, Yu LC, Chen HC, Chan CL. Comparison of models for the prediction of medical costs of spinal fusion in Taiwan Diagnosis-Related Groups by machine learning algorithms. Healthc Inform Res. 2018; 24(1):29–37.

11. Perkins NJ, Schisterman EF. The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol. 2006; 163(7):670–675.

12. Vincent JL. Endpoints in sepsis trials: more than just 28-day mortality? Crit Care Med. 2004; 32:5 Suppl. S209–S213.

13. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of clinical criteria for Sepsis: for the third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016; 315(8):762–774.

14. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA. 2017; 317(3):290–300.

15. Piri S, Delen D, Liu T, Zolbanin HM. A data analytics approach to building a clinical decision support system for diabetic retinopathy: developing and deploying a model ensemble. Decis Support Syst. 2017; 101:12–27.

16. Murphy KP. Naive Bayes classifiers. Vancouver, Canada: University of British Columbia;2006.

17. Amland RC, Sutariya BB. Quick Sequential [Sepsis-Related] Organ Failure Assessment (qSOFA) and St.John Sepsis Surveillance Agent to detect patients at risk of sepsis: an observational cohort study. Am J Med Qual. 2018; 33(1):50–57.

18. Azur MJ, Stuart EA, Frangakis C, Leaf PJ. Multiple imputation by chained equations: what is it and how does it work? Int J Methods Psychiatr Res. 2011; 20(1):40–49.

19. Milton JS, Arnold JC. Introduction to probability and statistics: principles and applications for engineering and the computing sciences. 4th ed. New York (NY): McGraw-Hill;2003.

20. Young PJ, Saxena M, Beasley R, Bellomo R, Bailey M, Pilcher D, et al. Early peak temperature and mortality in critically ill patients with or without infection. Intensive Care Med. 2012; 38(3):437–444.

21. Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011; 57(5):500–504.

22. Askim A, Moser F, Gustad LT, Stene H, Gundersen M, Asvold BO, et al. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality: a prospective study of patients admitted with infection to the emergency department. Scand J Trauma Resusc Emerg Med. 2017; 25:56.

23. Macdonald SP, Arendts G, Fatovich DM, Brown SG. Comparison of PIRO, SOFA, and MEDS scores for predicting mortality in emergency department patients with severe sepsis and septic shock. Acad Emerg Med. 2014; 21(11):1257–1263.

24. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, et al. Quick Sepsis-related Organ Failure Assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med. 2017; 195(7):906–911.

25. Rivers E, Nguyen B, Havstad S, Ressler J, Muzzin A, Knoblich B, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001; 345(19):1368–1377.

26. Ollendorf DA, Fendrick AM, Massey K, Williams GR, Oster G. Is sepsis accurately coded on hospital bills? Value Health. 2002; 5(2):79–81.

27. Sukanya C. Validity of principal diagnoses in discharge summaries and ICD-10 coding assessments based on national health data of Thailand. Healthc Inform Res. 2017; 23(4):293–303.