# LLM Codebook/Prompt Optimization

Boundaries of subjective categories such as "healthy food" shift with culture and person, yet conventional workflows demand stable labels from the start. Google Research's Agile Deliberation splits classifier building into concept scoping and concept iteration: users break a broad concept into sub-categories and explicit exclusions, then a vision-language model surfaces semantically borderline images for the user to judge, with feedback routed into an automatic prompt-optimization loop. Across 18 live 1.5-hour sessions, classifiers built this way substantially outperformed the compared standard baselines on the most challenging subjective images, and participants reported less fatigue and anxiety.

Boundaries of subjective categories such as "healthy food" shift with culture and person, yet conventional workflows demand stable labels from the start. Google Research's Agile Deliberation splits classifier building into concept scoping and concept iteration: users break a broad concept into sub-categories and explicit exclusions, then a vision-language model surfaces semantically borderline images for the user to judge, with feedback routed into an automatic prompt-optimization loop. Across 18 live 1.5-hour sessions, classifiers built this way substantially outperformed the compared standard baselines on the most challenging subjective images, and participants reported less fatigue and anxiety.

When large language models apply a codebook, a structured annotation guide that maps theoretical concepts onto labels, high aggregate scores can hide label-level failures. Across four datasets and five models, controlled prompt perturbations, repeated runs and a domain shift left worst-label F1 about 26 to 30 points below micro-F1, and models leaned on label names more than on definitions. These failures stayed invisible in accuracy and micro-F1 and persisted after prompt optimization.

When large language models apply a codebook, a structured annotation guide that maps theoretical concepts onto labels, high aggregate scores can hide label-level failures. Across four datasets and five models, controlled prompt perturbations, repeated runs and a domain shift left worst-label F1 about 26 to 30 points below micro-F1, and models leaned on label names more than on definitions. These failures stayed invisible in accuracy and micro-F1 and persisted after prompt optimization.