Sholem Hack - research works

GPT-4.0 demonstrated moderate performance (2.71 ± 1.46 overall), while Gemini 2.0 scored lowest (2.14 ± 1.01).

Mixed-effects modeling demonstrated significant group differences across all domains ( p ≤ 0.008).

Citation quality showed one of the largest between-group differences and strong reliability [ICC(2,1) = 0.68; ICC(2,7) = 0.94].

Manual verification of 123 AI-generated references revealed high citation accuracy for GPT-4.0 (89.6% fully accurate; 0% fabricated) compared with Gemini 2.0 (71.7% fully accurate; 23.9% fabricated).

Human-written abstracts were more often misclassified as AI than AI-generated abstracts were mistaken for human.

Human abstracts received significantly higher clarity and usefulness scores than LLM abstracts, though effect sizes were small.

Confidence did not correlate with correctness, indicating miscalibration of rater judgments.

Grok-generated abstracts were most easily identified as AI, whereas GPT-5 and Claude 3.5 more frequently resembled human writing.

Mean lay ratings for clarity across AI-generated forms were high overall.

However, when analyzed at the form level, differences between models were not statistically significant for clarity (mean difference 0.04; t (9) = -0.80; p = 0.44) or signing comfort (mean difference 0.13, t (9) = -1.68, p = 0.13).

Mean Flesch-Kincaid Grade Level was lower for AI-generated forms compared to official templates.

Although prompts targeted a 6th-8th grade reading level, achieved readability scores were slightly higher (8.8-9.4).

Twenty-four patients met inclusion criteria (mean age 45.2 years; 54.2% male).

All exhibited maxillary sinus atelectasis with orbital floor retraction.

No intraoperative complications or revision surgeries occurred.

Mean quality-of-life score was 4.58/5.

Differences were attenuated when vignettes contained clearer diagnostic information.

Other effects were smaller, including increased neuroimaging recommendations for low-income descriptors (+4.0 pp; 95% CI 2.1-5.8).

ConclusionLLM clinical recommendations varied by sociodemographic descriptors, particularly under diagnostic uncertainty.