Templated discharge notes made different patients look alike: pretraining only on narrative sections lifts ICD coding macro-F1 to 28.6%

Contrastive pretraining for automatic ICD coding no longer treats a whole discharge summary as one unit of similarity; the model reads only narrative sections such as chief complaint, history of present illness and past medical history. Earlier pretraining fed in templated sections such as hospital course and discharge diagnosis as well, so clinically distinct cases could be represented as overly similar simply because they shared a document template. On MIMIC-III-full, narrative-only pretraining gives PLM-CA 0.606 micro-F1 and 0.122 macro-F1 and PLM-ICD 0.601 and 0.118, while hospital-course-only pretraining drops PLM-ICD macro-F1 to 0.103.

Positive pairs are chosen by similarity over augmented ICD code sets rather than by whether the raw code strings overlap. Raw Jaccard similarity gives zero overlap for codes such as 251.1 and 251.2 that share no literal string yet are clinically related, so such pairs cannot be formed. Each document's label set is first extended with all ancestor codes in the ICD hierarchy before the Jaccard coefficient is computed, and pairs at or above a threshold enter pretraining; the threshold is 0.86 on MIMIC-III-full, 0.97 on MIMIC-IV ICD-9 and 0.82 on MIMIC-IV ICD-10.

ICD code descriptions are expanded by a large language model, encoded into label vectors, matched against document token representations by similarity, and concatenated as complementary features before the classification head. Earlier attempts to inject label descriptions were often tied to a specific architecture or relied on limited text that did not fully capture label semantics. On MIMIC-IV ICD-10 the module raises PLM-CA macro-F1 from 26.1% to 27.0% and PLM-ICD from 21.1% to 26.3%, with PLM-ICD P@8 rising from 69.9% to 71.5%; the interaction feature alone reaches only 0.089 macro-F1.

Predictions from the two backbones are merged by a mixture-of-experts module that weights them per document and per ICD code. PLM-CA wins more often on perinatal conditions and pregnancy complications, while PLM-ICD is stronger on circulatory and endocrine diseases, so neither backbone covers both. On MIMIC-IV ICD-10 linear weighted fusion gives 28.3% macro-F1 and 60.9% micro-F1, while the full system with mixture-of-experts fusion gives 28.6% and 61.1%.

Experiments cover three public benchmarks, MIMIC-III-full, MIMIC-IV ICD-9 and MIMIC-IV ICD-10; the narrative truncation applies only during pretraining, while fine-tuning uses full-length documents. The intended reader is a researcher working on clinical text representation learning or extreme multi-label classification.

Jaccard-guided pretraining relies on the quality and completeness of the ICD hierarchy, and some codes may have ambiguous parent-child relationships; the label-text module depends on LLM-generated expanded descriptions whose quality may vary, and it may not generalize to code sets without existing descriptions or to low-resource language settings. The authors also note that the current experiments provide no dedicated analysis of rare ICD codes and that the data come from a single institution.