A new way to probe personal agents' cross-service risks: sampling a tool-dependency graph reaches a 19.2% average harmful path rate

When a personal agent calls tools across services, a single benign request can produce harmful outcomes as the effects of its actions accumulate across services; X-AgentRed moves the discovery of such cross-service paths from LLM generation to sampling a tool-dependency graph. Earlier red-teaming methods either prompt an LLM to propose tool sequences from task and environment descriptions or decompose harmful objectives specified by humans or LLMs into seemingly benign action combinations, and their generation favors patterns learned during training, concentrating on familiar tool combinations. In a simulated ecosystem of 30 services and 914 tools, across the four target agents GPT-5.6 Sol, GLM-5.2, DeepSeek-V4-Pro, and Claude Opus 4.8, X-AgentRed reaches an average harmful path rate of 19.2%, against 10.5% for STAC and 2.7% for TRACE.

The framework first labels each tool with its effects, potential harms, and execution preconditions, then links these attributes into a directed graph so that one tool's effect can satisfy another tool's precondition. Sampling uses backward random walks that start from a tool associated with a target harm and follow incoming edges, then reverse the sampled sequence into a candidate recipe that is instantiated as a realistic scenario with a benign user request. Execution feedback drives two updates: paths that produce harm are diversified by replacing non-central tool nodes, while blocked paths have the preconditions at the blocked step refined and the preceding steps reconstructed; the authors manually inspected all 192 trajectories labeled harmful across the four target agents and found no evidence of reward hacking.

The results come from a multi-service ecosystem simulated with ToolSimulator, containing 30 services and 914 tools, with tool responses generated by Claude Opus 4.8; the four target agents are GPT-5.6 Sol, GLM-5.2, DeepSeek-V4-Pro, and Claude Opus 4.8, and each configuration is evaluated in one run within a fixed ecosystem. It suits readers focused on agent safety and tool-calling risk who want to understand how cross-service compositional risk is discovered.

The authors note that maintaining the HarmFlow graph incurs additional computational cost as services evolve, since new or modified tools require updating tool representations and re-evaluating affected dependencies. The judge sensitivity analysis shows that accepting either judge's assessment raises the harmful path rate relative to requiring agreement, and the authors state these comparisons measure sensitivity rather than judge accuracy, with the main results using unanimous agreement.