No weight updates: a customer-service agent tops the τ²-bench retail board with rules it learned from its own runs
Synopsis
IBM Research's ALTK-Evolve appends behavioral guidelines distilled from an agent's own trial-and-error trajectories to its system prompt, leaving model weights untouched. On the τ²-bench retail domain, an agent using the older non-reasoning model gpt-4.1 raises Pass4 from 57.5% to 67.5%, above every retail entry in the October 12, 2026 leaderboard snapshot and above RAFT-30B-A3B's 61.4% from fine-tuning on synthetic data. The same model extracts and applies the guidelines, with no human annotation.
Interpretation
A customer-service agent's reliability can improve without touching model weights: appending automatically distilled guidelines to the system prompt lifts the share of τ²-bench retail tasks solved on all four attempts from 57.5% to 67.5% with gpt-4.1. The usual alternative is fine-tuning on benchmark data, such as RAFT-30B-A3B trained on synthetic τ²-bench retail data, which reaches 61.4% Pass4. On 40 held-out retail test tasks the no-guideline baseline scores 57.5% and the full guideline set scores 67.5%, a 10-point gain; scored on the full base split the same configuration reaches 67.5%, above every retail entry in the leaderboard snapshot and 6.1 points above RAFT-30B-A3B.
The method hands the agent's successful and failed trajectories on training tasks to the same model, which distills them into short behavioral rules such as authenticating a customer first, getting explicit approval before database updates, and volunteering an itemized refund breakdown on returns. The rules are appended as a plain markdown list after the domain policy inside the system prompt, while the policy, tools and ReAct scaffold stay exactly as τ²-bench ships them. Runs use τ²-bench's own ReAct agent with a two-phase THINK→ACT loop, gpt-4.1 as the agent and gpt-5.5 as the simulated customer, four trials per task; guidelines mined on training tasks are applied unchanged to the held-out test set.
Perspective
Results cover the retail and airline customer-service domains of τ²-bench, evaluated on the benchmark's stock ReAct scaffold with a simulated customer, using gpt-4.1 and gpt-5.5 as agent models. The held-out test sets hold 40 retail and 20 airline tasks, and leaderboard scores are computed on the full base split, so they differ from the held-out test numbers.
More guidelines is not always better: in the airline domain 14 high-support guidelines lift Pass4 to 45% while the full set reaches only 40%, even though the full set has higher mean accuracy. With gpt-5.5 no extracted guideline reached support 3, so the high-support core had to use support 2. Guideline cost lands on input tokens, with the full retail set raising cost per conversation from about $0.084 to about $0.139.