Tool retrieval stops trusting semantics alone: execution feedback cuts fault selection from 86.2% to 2.9%
Lead
Instead of ranking tools by semantic similarity alone, an agent re-ranks candidates from each execution outcome, cutting selection of failed tools from 86.2% to 2.9% under 30% failure injection for about 0.172 ms per step.
Story
Tool retrieval becomes a decision that adjusts online with execution outcomes: the system re-ranks semantic candidates by whether a tool is currently working, so it can steer away from tools that fail at runtime. Earlier retrievers mapped queries to tool documentation by semantic relevance and treated retrieval as a static ranking task, largely ignoring whether a tool is available during execution. Under 30% failure injection, RATED lowers the fault selection rate to 2.9% against 86.2% for the ToolLLM API retriever; at 20% it reaches 2.5% against 18.1% for pretrained DRAGON.
Semantic matching and reliability estimation are split into two stages: a frozen encoder finds relevant tools, and a lightweight online module picks the one among the candidates that still works. Earlier work that folded execution feedback in still targeted the planner and updated the model from batched trajectories, so it could not reorder candidates between requests. The semantic encoder is contrastively fine-tuned on EcoAgent-Syn and then frozen, while re-ranking uses discounted LinUCB with a separate linear model per tool that down-weights older rewards before each update.
The added cost is small enough to sit inside an interactive agent loop. Batch GRPO-style updates need periodic policy synchronization and average 17.839 ms. On a single RTX 5070, encode-only averages 3.223 ms, while full RATED averages 3.395 ms and 4.391 ms at P95, 0.172 ms above encode-only.
What to watch
The reported results cover offline retrieval and injected-failure evaluation on the EcoAgent benchmark: 5,172 real tools drawn from LangChain, CrewAI, LlamaIndex and n8n, plus 6,379 queries and 9,946 verified query-tool pairs on the EcoAgent-Real test split. Dynamic metrics come from simulated persistent outages in which the failing set is fixed before the run and its tools stay unavailable from failure onset to the end. The intended readers are researchers and engineers working on the reliability of agent tool calls.
Offline retrieval and dynamic reliability do not move together: on EcoAgent-Real RATED improves MRR by 52.2% over the ToolLLM API retriever but stays below Ada-002 and SPLADE-v3, so high ranking accuracy on real queries does not guarantee robust tool selection when tools fail. Discounted LinUCB records 0 adaptation steps at 30% failure and 1,603 at 20%, while health check blacklisting fails to recover at 30%. Evaluating the cold-start path on newly added tools and validating RATED in live agent runtimes remain future work.