Evaluating Clinical NLP Services for Chest Radiograph Report Labeling: A Comparative Study on an Independent Pediatric Dataset.

Entities from impressions were mapped to 12 disease categories plus No Findings using regular expressions.

Significant differences were observed in the mean number of extracted entities among the NLP models (p < 0.001).

Assertion distributions also differed significantly (p < 0.001).

For the CheXpert labels comparison, the mean Fleiss' Kappa across all assertion categories was 0.68 ± 0.17, including all reports, and 0.35 ± 0.14, excluding reports where all six models predicted the disease as absent.