Low-bit agents stop dropping task steps: SteerDistill's closed-loop teacher corrections land within one point of BF16 at W4A16
Synopsis
Low-bit quantization cuts the inference cost of LLM agents but often makes them overlook task requirements or environmental conditions and take wrong steps in tool use. SteerDistill has a full-precision teacher locate the student's first behavioral divergence, issue a structured correction prompt, and refine that prompt using execution feedback until the revision passes local verification. The student then learns these corrections without guidance, and across five agentic benchmarks and three models under W4A16 its average score exceeds the strongest quantized baseline by 1.67–2.18 points and stays within one point of BF16.
Interpretation
Quantized agents miss task requirements or environmental conditions in tool-use tasks, for example reading a file before entering its directory or creating files but leaving them empty, so their environment transitions drift from successful full-precision trajectories. Earlier capability recovery for quantized models relied on offline data or the student's own on-policy rollouts; offline data may miss the student's failure contexts, and the student's rollouts rarely contain correct actions in those contexts. In a BFCL V4 multi-turn diagnostic, unguided student retries produced a correct action at only 9.6% of divergent turns, while teacher steering raised turn-level Pass@4 to 82.3%.
During training the full-precision teacher names the overlooked condition and supplies a corrective rule, the student answers again from the restored pre-error state, and a revision is kept only after its execution matches the successful reference. The teacher's prompt uses a structured format that separately states the discrepancy, the applicable conditions and the corrective rule, rather than free-form text. In the BFCL V4 ablation on Qwen3-8B under W4A16, the structured prompt scores 2.06 points above free-form, and the full-precision teacher as prompt source gains 5.10 points over the student and 1.57 points over GPT-5.6-Sol.
Perspective
The demonstrated scope is five agentic benchmarks (BFCL V4, -Bench, WebShop, ALFWorld and Search QA) and three models (Qwen3-8B, Qwen3.5-4B and Gemma 4 12B-it), mainly under W4A16, with W4A8 and W2A16 additionally tested on Qwen3-8B; inference uses only the trained quantized student, without the teacher or steering prompts.
Teacher-steered trajectory revision requires a successful teacher reference for the same task and initial state, restoration of the pre-error history and environment state, and reliable local verification; failed rollouts without a successful reference are retained for distillation but receive no steered corrections. Local verification checks a revision's effect in the current context and does not guarantee eventual task success or the correctness of every reasoning step in the response.