Psychometric characterization of human and artificial intelligence performance on cardiology residency in-service examination items.

Three frontier models were compared with two open-source comparators using a standardized zero-shot, repeated-query protocol and strict-majority scoring.

Across frontier models, performance increased progressively from hard to easy resident-derived item strata.

IRT-based re-analysis confirmed that higher latent item difficulty was independently associated with lower frontier-model accuracy.

These findings show that item-level psychometric analysis helps explain variation in frontier large language models performance beyond overall accuracy alone.