Agents stop guessing when to ask for help: ARIA picks consultation points and recovers 68% of the gap on SWE-bench at under half the cost

An executing agent no longer decides on its own when to ask a stronger model for help; a separately learned binary policy decides at which workflow contact points the advisor is brought in. Earlier routing, cascading and adaptive-computation methods allocate capability before execution or at task level, while fixed and prompted consultation leaves the timing to the executor itself, and language models struggle to localize their own reasoning failures and are poorly calibrated in confidence. On SWE-bench Verified, Terminal-Bench 2.1 and the telecom customer-service benchmark, ARIA recovers 68.2%, 44.4% and 80.6% of the executor–advisor success gap while spending 48.1%, 40.9% and 34.5% of Advisor-only evaluation cost.

The controller reads only signals the workflow already exposes, compressing the trajectory into 24 states and choosing Silent or Consult at each contact point. The state combines temporal position, execution evidence and whether advice has already entered the trajectory; it uses no model-internal confidence and fine-tunes neither language model. Action values are learned by every-visit tabular Monte Carlo control with sample-average updates from cost-penalized terminal outcomes, trained under an ε-greedy behavior policy and frozen to a greedy policy at deployment, where ties and unvisited states resolve to Silent.

A single consultation price governs the success–cost trade-off, and changing it requires no retraining of either model. The default policy attains the highest excess recovery on SWE-bench, 0.37, and raising the price makes the policy consult less often. Raising the price from the default cuts SWE-bench consultations from 2.78 to 2.03 and 1.55 per episode and evaluation cost from $22.09 to $18.78 and $16.63, while success falls from 48.0% to 34.0% and 29.3% and excess recovery falls from 0.37 to 0.15 and 0.11.

The policy targets executor–advisor agentic workflows and is evaluated on SWE-bench Verified, Terminal-Bench 2.1 and a telecom customer-service benchmark with one fixed model pair and the Magentic-One workflow, using development tasks disjoint from held-out evaluation tasks. The controller assumes only recurring decision points, so other single- or multi-agent systems can instantiate the same abstraction with their own recurring events. The authors present ARIA as a testbed for studying other model pairs, agent configurations and orchestration frameworks.

The authors report that gains are largest when similar decision points repeat across tasks and smaller on heterogeneous tasks such as Terminal-Bench, suggesting consultation policies are most useful when learned within related workflows. Experiments cover a single model pair and the Magentic-One framework. The failure analysis finds ignored agent input to be ARIA's dominant residual mode on SWE-bench, 58% of its failures, indicating that placement and advice uptake are distinct problems.