Iterate on a frontier model, serve on an 8B: distilled scheduler agrees with production records 0.913 of the time, beating its own teacher
Synopsis
A home-care scheduling agent no longer asks a small model to handle a narrow task cold. Zero-shot small models failed it, and retraining on every behavior change was too slow, so the behavior was first iterated in prompt space on GPT-5.4-nano, then 11.6k recorded production decisions were LoRA-tuned into Qwen3-8B. That model agrees with recorded decisions 0.913 of the time, above the teacher's 72.7% self-agreement when re-run on its own prompts.
Interpretation
When a small model takes over a narrow agent subtask, the behavior can be settled on a large model first and then moved wholesale to the small one, instead of retraining the small model after every change. Zero-shot small models failed this specific production task outright: five off-the-shelf small models fell short of the larger model's quality, and base Qwen3-8B produced malformed tool calls in 23% of cases. On per-caregiver outreach decisions for home-care shift filling, the tuned Qwen3-8B reached 0.913 agreement with held-out production decisions (95% CI 0.89–0.93), while the teacher re-run on its own recorded prompts agreed with itself only 72.7% of the time (95% CI 0.65–0.79).
The training targets are real production records rather than synthetic imitations of them. The dominant distillation approach trains on synthetic teacher-generated trajectories, so the training distribution differs from the serving distribution. Every production decision is durably recorded, and an extractor replays each batch through the same prompt builder and serializer the live evaluator uses, producing the identical chat context and pairing it with the recorded tool call; the corpus is 549 production batches with 11,621 training and 1,604 validation examples across 17 organizations.
In a blinded evaluation, a professional scheduler sided with the tuned small model more often than with the production decision on cases where the two disagreed. Measured only against the teacher's recorded labels, the student's false-approve rate looked flat, hiding any directional difference. The 192-case blinded evaluation included all 77 student-teacher disagreements plus stratified samples of 49 agreeing rejects and 66 agreeing approves; on the 77 disagreements the human sided with the tuned 8B 48 times versus 29 for the production decision (62% to 38%, 95% CI 0.51–0.72).
Perspective
The results concern one subtask in home-care agency scheduling: a yes/no outreach decision for each caregiver-shift pair, with up to 1,000 concurrent decisions per outreach. The evaluated models are LoRA-tuned Qwen3-4B and Qwen3-8B, with GPT-5.4-nano at high reasoning effort as the teacher. All numbers are offline replays, and the distilled model was not yet serving production traffic at the time of writing; the authors propose an online A/B test in production to account for prompt-cache dynamics, traffic mix, and downstream scheduler behavior.
The authors state that every result is an offline replay and that the true quality test is an online A/B test in production. Human ground-truth labels come from a single professional scheduler, and the statistics do not model correlation between cases from the same organization or batch. Results cover one subtask, one domain, and one teacher, so further work is needed to show the method works across a wide range of engineering tasks. The 4B model's 12% malformed-output rate leads the authors to place the capability floor between 4B and 8B parameters.