ToolSpace rebuilds agent training environments from real tool-call traces, lifting a 35B model's average score from 23.0 to 41.0 across seven benchmarks
Synopsis
ToolSpace reverses the order in which agent training environments are synthesized: it first mines data, state and retry dependencies between adjacent calls in execution traces from real MCP services and open agent corpora using deterministic rules, clusters the resulting weighted tool-composition graph into environments, synthesizes databases only where state is mutated, and then runs state-aware random walks with rollback to decompile replayable chains into natural-language tasks with assertion-based checks. Across seven long-horizon tool-use benchmarks, Qwen3.6-35B-A3B rises from 23.0 to 41.0 average score, or 43.2 with agent RL, and GLM-5.2 rises from 44.6 to 56.6.
Interpretation
The order of agent data synthesis is inverted: instead of building a database and scenario first and deriving tools from them, tools become the atomic unit of capability growth, real connections among tools are reconstructed first, and task complexity grows out of an expanding tool-composition space. Earlier environment-scaling methods are data-centric: they synthesize an underlying database and thematic scenario, then derive tools and tasks, so tool interfaces may diverge from real definitions, tool compositions lack execution grounding, and tasks rarely span more than one environment. The reconstructed tool graph holds 77,760 tool-call-pattern nodes drawn from 2,042 distinct canonical tools, with mean outdegree 2.3 and maximum 56; data, state and retry edges account for 69%, 25% and 6% respectively, and every edge is witnessed by adjacent calls in real trajectories.
The tool space grows directly out of execution traces: dependencies between adjacent calls are decided by deterministic rules rather than LLM judgment, and tools are then clustered into environments by connection density. A data dependency requires downstream arguments to be traceable to an upstream output with matching field types; a state dependency requires the first call to mutate an environment object that the second reads or further mutates; a retry dependency records an error followed by a tool switch and a success. Candidate edges are weighted by observed transition frequency and low-frequency edges are removed. The reference inventory covers 804 real MCP environments with an average of 10.86 matched tools per environment and a maximum of 436; 453 tools (22.2%) are operational and 181 environments (22.5%) are operational, while the rest replay recorded returns instead of synthesizing a database.
Tasks are reverse-written from executable chains: state-aware random walks on the weighted tool graph sample state-compatible successors by edge weight, roll back to a snapshot and resample on failure, and each retained chain is decompiled into a natural-language task that names neither tools nor call order. Each chain carries assertion-based verification scripts that check the final state, key intermediate snapshots and fields required in returned results, and only samples that replay from the initial state and pass every check enter the dataset. The pipeline synthesizes 34,376 trajectories, of which 21,385 have at least five tool calls; these average 18.86 tool calls with a maximum of 209, span 2.29 environments on average with a maximum of 22, and at least 66.2% involve two or more environments.
The same pipeline can harden existing benchmarks: anchored to AutomationBench and -Bench Banking, it yields AutomationBench Pro with 300 tasks and -Bench Banking Pro with 98 tasks. The Pro versions exceed the standard splits in tool-call turns and cross-environment breadth; on AutomationBench, average tool calls per model rise from 26.8–43.6 under Standard to 56.3–60.6 under Pro. All seven frontier models drop substantially on the Pro splits, with the largest AutomationBench drop reaching 38.2 points and Claude Opus 5.5 and GPT 6.1 Sol falling by 32.9 and 26.1 points; candidate tasks were independently reviewed over two days by three doctoral-level experts, and only tasks approved by at least two were retained.
Perspective
The demonstrated scope is evaluation on seven long-horizon tool-use benchmarks and construction of Pro versions of AutomationBench and -Bench Banking; training uses three backbones, Qwen3-8B, Qwen3.6-35B-A3B and GLM-5.2-744B-A40B, and all ToolSpace results are averaged over three independent runs.
Environment-synthesis savings are estimated with the number of synthesized files as a proxy: across 500 environments randomly sampled from Agent-World's public library, Agent-World synthesizes all 2,724 files while ToolSpace synthesizes only the 1,218 files tied to operation-bearing environments, 1,506 fewer or a 55.3% reduction; the source states this proxy does not directly measure token, wall-clock, API or compute cost. In the data-scaling analysis, BFCL V4 and OfficeQA Pro dip slightly below the backbone at the smallest SFT fraction, and the trend stabilizes only as the fraction increases.