J Korean Med Sci.  2021 Aug;36(31):e198. 10.3346/jkms.2021.36.e198.

Machine Learning Approach for Active Vaccine Safety Monitoring

• Kim Y
• Jang JH
• Park N
• Jeong NY
• Lim E
• Kim S
• Choi NK
• Yoon D

• KMID: 2519199
• DOI: http://doi.org/10.3346/jkms.2021.36.e198

Abstract

Background
Vaccine safety surveillance is important because it is related to vaccine hesitancy, which affects vaccination rate. To increase confidence in vaccination, the active monitoring of vaccine adverse events is important. For effective active surveillance, we developed and verified a machine learning-based active surveillance system using national claim data.
Methods
We used two databases, one from the Korea Disease Control and Prevention Agency, which contains flu vaccination records for the elderly, and another from the National Health Insurance Service, which contains the claim data of vaccinated people. We developed a casecrossover design based machine learning model to predict the health outcome of interest events (anaphylaxis and agranulocytosis) using a random forest. Feature importance values were evaluated to determine candidate associations with each outcome. We investigated the relationship of the features to each event via a literature review, comparison with the Side Effect Resource, and using the Local Interpretable Model-agnostic Explanation method.
Results
The trained model predicted each health outcome of interest with a high accuracy (approximately 70%). We found literature supporting our results, and most of the important drug-related features were listed in the Side Effect Resource database as inducing the health outcome of interest. For anaphylaxis, flu vaccination ranked high in our feature importance analysis and had a positive association in Local Interpretable Model-Agnostic Explanation analysis. Although the feature importance of vaccination was lower for agranulocytosis, it also had a positive relationship in the Local Interpretable Model-Agnostic Explanation analysis.
Conclusion
We developed a machine learning-based active surveillance system for detecting possible factors that can induce adverse events using health claim and vaccination databases. The results of the study demonstrated a potentially useful application of two linked national health record databases. Our model can contribute to the establishment of a system for conducting active surveillance on vaccination.

Keyword

Vaccines; Adverse Effects; Postmarketing Product Surveillance; Machine Learning; Cross-over Studies

Figure

• 

  Fig. 1 Flow chart of the study. (A) Anaphylaxis. (B) Agranulocytosis. Among the whole population, the numbers of people who were diagnosed with anaphylaxis or agranulocytosis at least once were 15,015 and 30,223, respectively. We selected people who did not have ruled-out diagnoses and for whom demographic information was available. After adapting exclusion criteria, the final sample size was 14,094 for anaphylaxis and 29,481 for agranulocytosis.

• 

  Fig. 2 Research workflow for developing an active surveillance system. First, total HOI diagnosis dates were extracted using the NHIS claim dataset. The period of 14 days before the HOI diagnosis was set as a risk window. The control window was randomly selected for 14 days excluding the risk window and washout period. If a HOI occurred several times, the HOI that re-occurred within 31 days was considered as the same and a continuous event of the previous HOI. In order to ensure independence between HOIs, a risk window was defined only for recurrent HOI events where the interval between HOIs was more than 6 months apart. Second, the prescription and vaccination information that occurred in each window was collected. If there was no prescription and vaccination information in the window, the window was removed. At the final step, the HOI prediction model was learned using the prescription and vaccination information in each window. After that, using the feature importance ratio and LIME analysis of the model, a suspected drug or vaccine that could cause the HOI was determined.FI = feature importance, HOI = health outcome of interest, KDCA = Korea Disease Control and Prevention Agency, LIME = Local interpretable Model-agnostic Explanation, NHIS = National Health Insurance Service, SIDER = Side Effect Resource database.

• 

  Fig. 3 Comparison of SIDER-listed features and features with high importance ratios. (A) Anaphylaxis. (B) Agranulocytosis. The Venn diagrams show the number of features in each result, and the intersection includes those features that are not only already listed in the SIDER database but also have an importance ratio of over 1 (left) or 2 (right).GORD = gastro-esophageal reflux disease, NSAID = nonsteroidal anti-inflammatory drug, SIDER = Side Effect Resource database.

• 

  Fig. 4 Comparison of LIME positive features and features with an importance ratio of over 2. The Venn diagrams show the number of features in each result and the intersection includes those features that are not only LIME positive but also have an importance ratio of over 2 (A) for anaphylaxis and (B) for agranulocytosis.GORD = gastro-esophageal reflux disease, LIME = Local interpretable Model-agnostic Explanation.

Reference

1. Rubin R. Difficult to determine herd immunity threshold for COVID-19. JAMA. 2020; 324(8):732.

2. Jung J. Preparing for the coronavirus disease (COVID-19) vaccination: evidence, plans, and implications. J Korean Med Sci. 2021; 36(7):e59. PMID: 33619920.

3. Salmon DA, Dudley MZ, Glanz JM, Omer SB. Vaccine hesitancy: causes, consequences, and a call to action. Vaccine. 2015; 33(Suppl 4):D66–71. PMID: 26615171.

4. Loomba S, de Figueiredo A, Piatek SJ, de Graaf K, Larson HJ. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat Hum Behav. 2021; 5(3):337–348. PMID: 33547453.

5. Dubé E, Laberge C, Guay M, Bramadat P, Roy R, Bettinger J. Vaccine hesitancy: an overview. Hum Vaccin Immunother. 2013; 9(8):1763–1773. PMID: 23584253.

6. Takahashi H, Pool V, Tsai TF, Chen RT. The VAERS Working Group. Adverse events after Japanese encephalitis vaccination: review of post-marketing surveillance data from Japan and the United States. Vaccine. 2000; 18(26):2963–2969. PMID: 10825597.

7. Jeong NY, Park S, Lim E, Choi NK. An introduction of the active vaccine safety surveillance system in foreign countries. J Health Info Stat. 2019; 44(4):317–330.

8. Choe YJ, Bae GR. Management of vaccine safety in Korea. Clin Exp Vaccine Res. 2013; 2(1):40–45. PMID: 23596589.

9. Davis RL, Kolczak M, Lewis E, Nordin J, Goodman M, Shay DK, et al. Active surveillance of vaccine safety: a system to detect early signs of adverse events. Epidemiology. 2005; 16(3):336–341. PMID: 15824549.

10. Yih WK, Kulldorff M, Fireman BH, Shui IM, Lewis EM, Klein NP, et al. Active surveillance for adverse events: the experience of the Vaccine Safety Datalink project. Pediatrics. 2011; 127(Suppl 1):S54–64. PMID: 21502252.

11. Brown JS, Kulldorff M, Chan KA, Davis RL, Graham D, Pettus PT, et al. Early detection of adverse drug events within population-based health networks: application of sequential testing methods. Pharmacoepidemiol Drug Saf. 2007; 16(12):1275–1284. PMID: 17955500.

12. Huh K, Kim YE, Radnaabaatar M, Lee DH, Kim DW, Shin SA, et al. Estimating baseline incidence of conditions potentially associated with vaccine adverse events: a call for surveillance system using the Korean National Health Insurance Claims Data. J Korean Med Sci. 2021; 36(9):e67. PMID: 33686812.

13. Lee GM, Romero JR, Bell BP. Postapproval Vaccine Safety Surveillance for COVID-19 Vaccines in the US. JAMA. 2020; 324(19):1937–1938. PMID: 33064152.

14. Huang YL, Moon J, Segal JB. A comparison of active adverse event surveillance systems worldwide. Drug Saf. 2014; 37(8):581–596. PMID: 25022829.

15. Nikfarjam A, Sarker A, O’Connor K, Ginn R, Gonzalez G. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. J Am Med Inform Assoc. 2015; 22(3):671–681. PMID: 25755127.

16. Botsis T, Nguyen MD, Woo EJ, Markatou M, Ball R. Text mining for the Vaccine Adverse Event Reporting System: medical text classification using informative feature selection. J Am Med Inform Assoc. 2011; 18(5):631–638. PMID: 21709163.

17. Jeon E, Kim Y, Park H, Park RW, Shin H, Park HA. Analysis of adverse drug reactions identified in nursing notes using reinforcement learning. Healthc Inform Res. 2020; 26(2):104–111. PMID: 32547807.

18. Lindquist M, Ståhl M, Bate A, Edwards IR, Meyboom RH. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000; 23(6):533–542. PMID: 11144660.

19. Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med. 2012; 4(125):125ra31.

20. Wang SV, Gagne JJ, Maro JC, Eworuke E, Kattinakere S, Kulldorff M, editors. Development and Evaluation of a Global Propensity Score for Data Mining with Tree-Based Scan Statistics. Sentinel;2018.

21. Choi B, Kim SH, Lee H. Are registration of disease codes for adult anaphylaxis accurate in the emergency department? Allergy Asthma Immunol Res. 2018; 10(2):137–143. PMID: 29411554.

22. Helgeland J, Tomic O, Hansen TM, Kristoffersen DT, Hassani S, Lindahl AK. Postoperative wound dehiscence after laparotomy: a useful healthcare quality indicator? A cohort study based on Norwegian hospital administrative data. BMJ Open. 2019; 9(4):e026422.

23. Ribeiro MT, Singh S, Guestrin C. “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016. p. 1135–1144.

24. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine learning in Python. Journal of machine Learning research. 2011; 12:2825–2830.

25. Regateiro FS, Marques ML, Gomes ER. Drug-induced anaphylaxis: an update on epidemiology and risk factors. Int Arch Allergy Immunol. 2020; 181(7):481–487. PMID: 32396909.

26. Montañez MI, Mayorga C, Bogas G, Barrionuevo E, Fernandez-Santamaria R, Martin-Serrano A, et al. Epidemiology, mechanisms, and diagnosis of drug-induced anaphylaxis. Front Immunol. 2017; 8:614. PMID: 28611774.

27. Schweizer MT, Huang P, Kattan MW, Kibel AS, de Wit R, Sternberg CN, et al. Adjuvant leuprolide with or without docetaxel in patients with high-risk prostate cancer after radical prostatectomy (TAX-3501). Cancer. 2013; 119(20):3610–3618. PMID: 23943299.

28. Tse SS, Kish T. Octreotide-associated neutropenia. Pharmacotherapy. 2017; 37(6):e32–7. PMID: 28488730.

29. Kim MJ, Shim DH, Cha HR, Kim CB, Kim SY, Park JH, et al. Delayed-onset anaphylaxis caused by IgE response to influenza vaccination. Allergy Asthma Immunol Res. 2020; 12(2):359–363. PMID: 32009327.

30. Couronné R, Probst P, Boulesteix AL. Random forest versus logistic regression: a large-scale benchmark experiment. BMC Bioinformatics. 2018; 19(1):270. PMID: 30016950.

31. FDA. Ketorolac tromethamine. Updated 2014. Accessed March 3, 2021. https://www.accessdata.fda.gov/drugsatfda_docs/label/2014/074802s038lbl.pdf.

32. Dhakal OP, Dhakal M, Bhandari D. Domperidone-induced dystonia: a rare and troublesome complication. BMJ Case Rep. 2014; 2014:bcr2013200282.

33. Kaplan SA, Chughtai BI. Safety of tamsulosin: a systematic review of randomized trials with a focus on women and children. Drug Saf. 2018; 41(9):835–842. PMID: 29737501.