High scores, hidden failures: testing codebook-based LLM coding across four datasets and five models
When large language models apply a codebook, a structured annotation guide that maps theoretical concepts onto labels, high aggregate scores can hide label-level failures. Across four datasets and five models, controlled prompt perturbations, repeated runs and a domain shift left worst-label F1 about 26 to 30 points below micro-F1, and models leaned on label names more than on definitions. These failures stayed invisible in accuracy and micro-F1 and persisted after prompt optimization.