Tool retrieval stops trusting semantics alone: execution feedback cuts fault selection from 86.2% to 2.9%
Synopsis
When an AI agent calls an external tool, retrieval previously ranked candidates by semantic relevance alone and could not tell whether a tool would actually work at runtime. RATED pairs a frozen dual-encoder for semantic matching with a lightweight contextual bandit that re-ranks candidates from each execution's success or failure feedback. Under 30% failure injection, fault selection drops from 86.2% for the ToolLLM API retriever to 2.9%, at 0.172 ms added latency per step.
Interpretation
Tool selection for agents now accounts for whether a tool is currently usable, not only whether it looks semantically similar to the query. Earlier tool retrieval treated the task as static ranking, mapping queries to tool documentation by semantic relevance while runtime availability stayed outside the ordering. Under 30% failure injection, fault selection falls from 86.2% for the ToolLLM API retriever to 2.9%, while post-failure success rate stays above 82%.
The design splits semantic matching from reliability estimation, keeping the first stage fixed offline and updating the second online. The semantic stage is a DRAGON dual-encoder contrastively fine-tuned on synthetic queries and then frozen; the reliability stage is a discounted LinUCB contextual bandit that keeps one linear model per tool conditioned on the query context. Evaluation runs on EcoAgent, a benchmark aggregating 5,172 real tools from four open-source agent platforms and over 210,000 query-tool pairs, with dynamic experiments on the EcoAgent-Real test split.
Perspective
The reported scope is offline retrieval evaluation plus sequential selection experiments under injected failures, with the tool repository drawn from four open-source platforms: LangChain, CrewAI, LlamaIndex, and n8n. It suits researchers and engineers working on agent tool calling and retrieval systems. The source lists evaluating the cold-start path for newly added tools and validating RATED in live agent runtimes as future work.
On EcoAgent-Real, RATED's offline MRR stays below Ada-002 and SPLADE-v3, so ranking accuracy on real queries is not the advantage. The dynamic results come from simulated persistent outages whose failing set is fixed before each run, not from real API interruptions. Evaluation of the cold-start path for new tools and validation in live runtimes remain open.