Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation

Radiology reports convey detailed clinical observations and capture diagnostic reasoning that evolves over time. However, existing evaluation methods are limited to single-report settings and rely on coarse metrics that fail to capture fine-grained clinical semantics and temporal dependencies. We introduce **LUNGUAGE** , a benchmark dataset of structured radiology reports that serves as a gold standard for evaluating structured report frameworks. It is designed to support comprehensive assessment of both single-report interpretation and longitudinal reasoning. Constructed from a subset of the MIMIC-CXR test set, LUNGUAGE comprises 1,473 chest X-ray reports from 230 patients, annotated with over 17,000 expert-verified entities and 23,000 relation-attribute pairs across 18 relation types. An additional subset of 80 sequential reports from 10 patients captures disease progression across 3 to 14 studies per patient, covering time intervals from 1 to 1,200 days. These are annotated with over 41,000 pairwise comparisons, grouped into semantically and temporally coherent groups. The dataset also includes a schema-aligned vocabulary covering diagnostic entities and attributes. All annotations were conducted and verified by board-certified radiologists, resulting in a clinically grounded resource for structured understanding and temporal reasoning in radiology.