Evaluating the diagnostic performance of a large language model-powered chatbot for providing…

AI / informaticsJournal of cutaneous pathology· Sep 2024

Evaluating the diagnostic performance of a large language model-powered chatbot for providing immunohistochemistry recommendations in dermatopathology.

Myles R McCrary, Justine Galambus, Wei-Shen Chen

DermatopathologyAI / informaticsQ2IF 1.3

Chatbot immunohistochemistry recommendations for skin diseases were factually incorrect 13.9% of the time

Evaluating the diagnostic performance of a large language model-powered chatbot for providing immunohistochemistry recommendations in dermatopathology.

Myles R McCrary … Wei-Shen Chen

Journal of cutaneous pathology · Sep 2024 · Q2 in Pathology

Background

Large language model (large language model)-powered chatbots such as ChatGPT have numerous applications.

Methods

We queried a commonly used chatbot for the immunophenotypes of 51 cutaneous diseases, including a diverse variety of epidermal, adnexal, hematolymphoid, and soft tissue entities.

n = 51 cutaneous diseases

13.9%

Results

13.9%

chatbot IHC recommendations were wrong for more than 1 in 10 diagnoses

n = 51 cutaneous diseases

More results

Clustering analysis demonstrated that recommendations correlated with tumor type, suggesting chatbots can supply appropriate panels.

Citations were rarely clinically useful (24.5%).

Many were confabulated (27.2%).

More results

Prompt responses for cutaneous adnexal lesions tended to be less accurate while literature references were less useful.

“

Conclusion · 1 of 2

This foundational study suggests that large language model-powered chatbots may be useful for generating immunohistochemical panels for dermatologic diagnoses. However, specific performance capabilities and biases must be considered.

Conclusion · 2 of 2

In addition, extreme caution is advised regarding the tendencies to fabricate material. Future models intentionally fine-tuned to augment diagnostic medicine may prove to be valuable.

Evaluating the diagnostic performance of a large language model-powered chatbot for providing immunohistochemistry recommendations in dermatopathology.

Evaluating the diagnostic performance of a large language model-powered chatbot for providing immunohistochemistry recommendations in dermatopathology.

Related papers