Cheaper decisions, not always faster tasks: five agents and 4,500 runs draw the line

Handing an agent's action selection to a model built for fast structured judgments makes each decision much cheaper and quicker. The prior setup used one large model for both reasoning and action choice, paying full reasoning cost and latency at every step. Across five agent families, two benchmarks, and 4,500 task executions, Jev cut mean decision cost by 66.1% to 78.4% and decision time by 73.9% to 81.5%, with decision times falling from 4.13 to 8.13 seconds down to 1.04 to 1.75 seconds.

Savings at the decision level do not carry over to the task level on their own, because other parts of the workflow absorb them. Task-level gains are much smaller than decision-level gains and vary by workflow. On ALFWorld, ReAct and Reflexion cut task cost by 50.9% and 32.6%; on AndroidWorld, T3A, AutoDroid, and FastAgent cut cost by 17.1%, 68.8%, and 43.0%, while FastAgent's task time rose by 0.7% as its auxiliary calls and non-model execution grew longer.

Whether the task actually gets done decides what the savings are worth. Lower spending per attempt does not imply lower spending per successful outcome. On task-run pairs both configurations completed, ALFWorld kept 64.8% to 66.2% lower cost and 59.2% to 60.8% shorter execution; pooled over all attempts, T3A's cost per delivered success rose from 68.70 to 73.97 mUSD and FastAgent's time per delivered success rose from 239.4 to 377.8 seconds.

Failures usually come from an error that is never corrected, or from a task declared finished too early. Earlier evaluation looked only at final success rates, without labeling recovery and termination behavior in the execution traces. Among 57 Jev executions in persistent-loss pairs, 40 showed failed recovery and 29 ineffective search, against 3 and 2 of 57 baseline executions; Reflexion-J won 383/402 first trials versus the baseline's 376/402 but recovered only 11/19 initial failures against 25/26, ending with lower final success.

The study covers five agent families in ALFWorld household tasks and AndroidWorld mobile app use, both with text observations and bounded action interfaces. Each configuration runs three times on the same fixed tasks with a shared seed, and comparisons are within each family between the baseline and its Jev variant. ALFWorld adds an LLM-Choice control under a shared candidate interface, separating the interface effect from the decision-model effect. It suits readers weighing cost and reliability in agent workflows, particularly those considering a fast decision model for tool selection or workflow routing.

The authors state that findings are limited to the evaluated integrations, where candidate construction, auxiliary generation, and feedback mechanisms jointly shape performance, so not every difference can be attributed to Jev alone. Latency and cost depend on the evaluated model versions, service conditions, and experimental prices; hosted inference adds variation, and future model updates or price changes may alter the trade-offs. The three repetitions characterize execution variability on fixed task instances and do not cover newly sampled environments or task distributions. Costs are estimates under the stated accounting rules, and retry adjustments and imputed usage introduce measurement uncertainty where applicable.