Agent safety moves past web prompt injection: attacker and defender co-evolve across tool-use tasks
Synopsis
LLM agents complete tasks through multi-turn tool calls, but malicious content in user queries, tool descriptions, or environment observations can steer them into unsafe actions. AgentSafeEvo frames safety as an attacker-defender game spanning diverse tasks and attack types: the attacker learns to build tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under those attacks, with learnability-guided task weighting adjusting each task's contribution to both updates. On agent safety and general capability benchmarks, the method improves adversarial robustness while maintaining general agent capabilities.
Interpretation
Agent safety training moves from language-level safety alone to an attacker-defender game spanning diverse tasks and attack types. Earlier adversarial co-evolution work focused mainly on language-level safety or on indirect prompt injection in web tasks. On agent safety and general capability benchmarks, the method improves adversarial robustness while maintaining general agent capabilities.
Attacker and defender set problems for each other on the same tasks, with the attacker hunting for weak points in the defender's tool use. The attacker learns to construct adversarial agent tasks, while the defender learns to complete tasks safely under those attacks, forming an attacker-defender game. The framework is built on co-evolutionary reinforcement learning, with tasks and attack types spanning multiple categories.
Each task's contribution to the attacker and defender updates is no longer uniform, but weighted by the defender's reward variability within that task. Learnability-guided task weighting uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. This weighting mechanism is combined with attacker-defender co-evolutionary training for agent safety tasks.
Perspective
The work targets safety training for agents that need multi-turn tool calls, spans diverse tasks and attack types, and is evaluated on agent safety and general capability benchmarks. The summary does not name the benchmarks, task counts, or deployment settings, so the intended readers are researchers and engineers working on agent safety training and adversarial robustness.
The summary reports only that the method improves adversarial robustness while maintaining general capability on safety and general capability benchmarks, without specific metric values, benchmark names, or an attack-type list. Learnability-guided task weighting depends on within-task defender-reward variability, and the summary does not describe how it behaves across different task distributions.