Better task completion, more leakage: 936 executable agent tasks show 32.5% to 94.4% privacy leakage

Whether an agent passes sensitive user data to the wrong recipient while working in everyday applications can now be measured with executable tasks. Earlier evaluations mostly stayed at model-level privacy judgments, or tested single-step actions in a few application environments with manually constructed disclosure constraints. Across 936 tasks and 12 agents, single-run leakage ranges from 32.5% to 94.4%, and repeating each task five times raises leakage by 4 to 19 percentage points.

Each task carries an explicit privacy contract that names which information is protected and which external recipients must not receive it, and the tested agent never sees that contract. Protected attributes are drawn from a fixed list of 39 attribute types in 10 categories, and must pass unanimous agreement across three privacy personas plus validation against the corresponding policy text. The contracts are grounded in 7 privacy laws and regulations, and the tasks span 11 application environments including Gmail, Google Docs, Slack, Salesforce, Calendar and WhatsApp, with 39 environment combinations and an average of 2.3 environments per task.

Leakage is uneven: explicit identifiers are protected comparatively well, while contextual information such as preferences, booking details and messages is carried out more often. Personal preferences and booking details show average leakage rates of 84.7% and 76.5%, compared with 17.1% for credit-card information and 29.4% for SSNs. Attribute-level leakage rates show Calendar as the highest-leakage information source on average and WhatsApp as the highest-ranked information destination.

This evaluation is aimed at researchers and engineers studying agent privacy behavior, and covers 11 application environments, 936 executable tasks and 7 privacy laws, with all tasks and data synthetic. It measures leakage in a controlled environment rather than deployment behavior on real user data.

Synthetic data cannot fully capture the contextual complexity, ambiguity and personalized privacy expectations of real interactions; the evaluation currently covers single-agent settings, leaving inter-agent communication and information sharing out of scope; and the platform focuses on measuring leakage rather than developing and validating defenses.