High scores, hidden failures: testing codebook-based LLM coding across four datasets and five models
Synopsis
When large language models apply a codebook, a structured annotation guide that maps theoretical concepts onto labels, high aggregate scores can hide label-level failures. Across four datasets and five models, controlled prompt perturbations, repeated runs and a domain shift left worst-label F1 about 26 to 30 points below micro-F1, and models leaned on label names more than on definitions. These failures stayed invisible in accuracy and micro-F1 and persisted after prompt optimization.
Interpretation
A codebook-driven LLM annotation pipeline cannot be judged by aggregate scores alone: the same pipeline shows severe label-level imbalance and shifts with runs, domains and prompt formatting. Earlier work reported overall agreement with human annotations and strong predictive performance, and treated LLMs as scalable substitutes for human coders on that basis. On Minimum Wage, Value Creation Science and CCC, the gap between worst-label F1 and micro-F1 was 30, 26 and 28 points respectively, a difference entirely invisible in aggregate reporting.
Models rely heavily on label names themselves rather than on the definitions the codebook supplies. Codebook-based analysis assumes the model follows the provided definitions instead of defaulting to background knowledge from pretraining. A Label-Only setting that keeps label names but drops definitions still scored surprisingly high on Value Creation Science, while replacing labels with generic pseudo-labels or swapping names and definitions dropped performance substantially.
Handing the codebook to an LLM for automatic optimization amplifies label-level bias and makes results more dataset-dependent. Automatic prompt optimization and multi-agent frameworks iteratively rewrite prompts to raise predictive performance, but their effect on robustness and conceptual validity had not been systematically examined. Adding both positive and negative examples raised micro-F1, macro-F1, accuracy and macro-Precision on CCC and Value Creation Science, yet drove worst-label F1 down to 5.67 and 11.32; APO removed one label entirely on Value Creation Science.
Moving one codebook to another text type in the same topic raised per-label scores while the full correct label set was predicted for no instance at all. Codebook analysis assumes a coding framework stays valid across new data sources within the same theoretical domain. Applying the Value Creation Science codebook, built on academic abstracts, to news text on the same topics raised micro-F1 from 44.89 to 62.22 and macro-F1 from 39.57 to 59.55, while overall accuracy was 0.0.
Perspective
The findings rest on four datasets, five models and the corresponding codebook configurations, which the authors describe as reflecting common computational social science settings; the evaluation framework itself is described as applicable to any dataset, codebook or LLM. The authors hope future work extends it to larger, multilingual and domain-specific contexts.
The authors state that the analysis is limited to a small set of datasets, codebooks and model configurations, and that results may not generalize across domains, languages or annotation frameworks; the human evaluation of optimized codebooks relies on only two experts, which may limit intersubjective validity. Rating variance was low and Cohen's κ relatively low, although exact agreement was 51.6% and 87.3% of ratings differed by at most one scale point.