Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance…

AI / informaticsJournal of medical systems· Aug 2026

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection.

Andreas Praschk, Valentin Fischill-Neudeck, Thomas Caspari, Hans-Peter Wiesinger

Read paper

Large Language Models in CareAI / informaticsQ1IF 8.3

When both AI models misclassified vaccine stance, they nearly always made the identical error.

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection.

Andreas Praschk … Hans-Peter Wiesinger

Journal of medical systems · Aug 2026 · Q1 in Health Informatics

Purpose

This mixed-methods study assessed whether reasoning-enabled large language models (large language models) can classify stances towards COVID-19 vaccination on X (formerly Twitter) and whether model-generated chain-of-thought (chain-of-thought) summaries contain reasoning failures relevant to transparent and auditable public health applications.

Results

when both models erred, they almost always made the same mistake

Δmacro-F1 0.02 (95% CI 0.01 to 0.03)

null 0

0.01

0.03

CI excludes the null - significant

More results

We reported accuracy and macro-F1, measured chain-of-thought availability, and qualitatively analysed dual-error cases (tweets misclassified by both models) using Mayring's content analysis guided by the FUTURE-AI framework.

More results

Under the reasoning-intensive settings, chain-of-thought availability differed: Gemini returned a reasoning summary for all tweets, whereas o4-mini did so for 64.7%.

“

Conclusion

Chain-of-thought availability, label-rationale coherence, and safeguards against systematic reasoning failures offer candidate explainability-readiness metrics, alongside accuracy, for trustworthy digital epidemiology.

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection.

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection.

Related papers