Subjective image classifiers stop guessing labels: Agile Deliberation has users sharpen boundaries first, then optimizes prompts
Synopsis
Boundaries of subjective categories such as "healthy food" shift with culture and person, yet conventional workflows demand stable labels from the start. Google Research's Agile Deliberation splits classifier building into concept scoping and concept iteration: users break a broad concept into sub-categories and explicit exclusions, then a vision-language model surfaces semantically borderline images for the user to judge, with feedback routed into an automatic prompt-optimization loop. Across 18 live 1.5-hour sessions, classifiers built this way substantially outperformed the compared standard baselines on the most challenging subjective images, and participants reported less fatigue and anxiety.
Interpretation
Classifiers for subjective categories no longer require users to arrive with a complete definition; defining the concept itself becomes a back-and-forth conversation with the system. Traditional training workflows assume the annotator already holds a clear, stable definition of the category, forcing users to supply rigid labels even while their own understanding is still shifting. Evaluation used 18 live 1.5-hour sessions in which participants interacted with the system and alternative baselines in randomized order to prevent learning bias.
The system first breaks a broad concept into a structured hierarchy of sub-concepts and explicit exclusions, then has a vision-language model actively surface semantically borderline images for the user to adjudicate. This borrows active learning's idea of retrieving informative samples, but selects on whether an image challenges the current text definition rather than on machine uncertainty. For instance, the system may present a fruit smoothie, forcing the user to decide whether it counts as a healthy beverage for containing raw fruit or as an excluded processed food for its high sugar content.
User judgments feed directly into an automatic prompt-optimization loop, so the vision-language model's classification boundaries shift in tandem with the user's refined definitions. Users no longer hand-write and repeatedly tweak prompts; they simply give natural feedback on the borderline images the system surfaces. Participants reported pronounced cognitive fatigue from guessing search terms to find edge cases and anxiety that fixing one prompt might degrade parts already fixed; those using the workflow reported significantly less stress, frustration and mental drain, and every participant preferred it.
Perspective
Evaluation is confined to two highly subjective visual domains, compared against three standard paradigms through 18 live 1.5-hour sessions; the material gives no specific performance metric values and does not report behavior on objective categories or non-visual tasks.
The material says gains were most dramatic on the most challenging subjective images but reports no specific metrics; user preference and reduced fatigue come from participant self-report rather than independent measurement.