Ting Cao

Low-bit quantization cuts the inference cost of LLM agents but often makes them overlook task requirements or environmental conditions and take wrong steps in tool use. SteerDistill has a full-precision teacher locate the student's first behavioral divergence, issue a structured correction prompt, and refine that prompt using execution feedback until the revision passes local verification. The student then learns these corrections without guidance, and across five agentic benchmarks and three models under W4A16 its average score exceeds the strongest quantized baseline by 1.67–2.18 points and stays within one point of BF16.

Low-bit quantization cuts the inference cost of LLM agents but often makes them overlook task requirements or environmental conditions and take wrong steps in tool use. SteerDistill has a full-precision teacher locate the student's first behavioral divergence, issue a structured correction prompt, and refine that prompt using execution feedback until the revision passes local verification. The student then learns these corrections without guidance, and across five agentic benchmarks and three models under W4A16 its average score exceeds the strongest quantized baseline by 1.67–2.18 points and stays within one point of BF16.