J Retin.  2024 Nov;9(2):119-126. 10.21561/jor.2024.9.2.119.

Limitations of Obtaining Medical Information about Age-Related Macular Degeneration from Artificial Intelligence Chatbots

• Na DG
• Yoon YS
• Kim JH

• KMID: 2561873
• DOI: http://doi.org/10.21561/jor.2024.9.2.119

Abstract

Purpose
This study aims to evaluate the quality of and trends in artificial intelligence (AI) chatbot responses to questions related to age-related macular degeneration (AMD) and to analyze the frequency of incorrect key information.
Methods
Three chatbots, ChatGPT 3.5, ChatGPT 4.0, and Gemini, were used in this study. Nine questions were formulated covering general information about: 1) AMD, 2) AMD treatment options, 3) effects and side effects of intraocular injections. Each question was queried three times with each of the three chatbots using two different accounts. Responses to items 1–3 were rated as poor/acceptable/good, and the frequency of incorrect key information was noted.
Results
Overall, the majority of the queries received acceptable or good responses. Poor-quality responses were noted in 16.7% of ChatGPT 3.5 responses. Incorrect key information was present in 6.5% of all responses.
Conclusions
While AI chatbots generally provided acceptable responses to questions regarding AMD, some responses contained incorrect key information, suggesting the need for caution when accessing medical information through AI chatbots.

Keyword

Age-related macular degeneration; Artificial intelligence; Chatbot