Korean J Otorhinolaryngol-Head Neck Surg.  2024 Mar;67(3):146-151. 10.3342/kjorl-hns.2023.00451.

Can We Trust the Information on Allergic Rhinitis From Chat Generated Pre-Trained Transformer?

• Lee SJ
• Choi YS
• Bae CH
• Song SY
• Kim YD
• Na HG

• KMID: 2553734
• DOI: http://doi.org/10.3342/kjorl-hns.2023.00451

Abstract

Background and Objectives
Chat Generated Pre-Trained Transformer (ChatGPT) is a large language model, which allows consumers to get information with one simple question for free. There are few studies that reported the reliability and usefullness of ChatGPT in the field of otorhinolarynoglogy, so we would like to investigate the reliability of information about allergic rhinitis generated by ChatGPT. Materials and Method We asked ChatGPT 35 questions related to allergic rhinitis based on the textbook from Korean Society of Otorhinolaryngology-Head and Neck Surgery, the clinical guidelines of American Academy of Otolaryngology-Head and Neck Surgery, and the guidelines of Allergic Rhinitis and Its impact on Asthma, and recorded the generated answers. Five specialists from our department were shown the answers in a blind test to assess their reliability. Each question was rated either ‘accurate’ or ‘inaccurate,’ and if the contents were not found in the textbook or the guideline, it was rated as ‘unreliable.’
Results
Of the 35 questions, 26 (74%) were identified as ‘accurate,’ 9 (26%) were ‘inaccurate,’ and 0 (0%) were ‘unreliable.’ Questions about epidemiology, causes, diagnosis, and prognosis were found to be more accurate than the average, whereas definitions and treatments were less accurate than the average.
Conclusion
The information about allergic rhinitis generated by ChatGPT was quite reliable, showing that ChatGPT can be helpful in understanding and treating the disease. It is necessary to use the developing medical artificial intelligence wisely.

Keyword

Allergy and immunology; Artificial intelligence; ChatGPT; Data accuracy; Rhinitis, allergic