AI agents are graded on the tax question clients never ask; Claude Sonnet 5.5 passes 10.8% of 50 assignments

Tax evaluation for AI agents now covers a whole professional assignment, with the client request, the source records and the finished deliverable handed over together instead of a question with a known answer. Earlier tax evaluations generally start from a defined problem, such as a hard tax question with a grading rubric, a return to calculate against expected filing values, or a simulated taxpayer to interview for missing information; whether the client asked the right question was not part of the test. Scored as pass@1 against a 95% rubric threshold, Claude Sonnet 5.5 leads at a 10.8% pass rate, ahead of Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%.

Each environment supplies a client request plus a workspace of messy records, and the agent must decide what deserves investigation, reconcile conflicting evidence and produce a return, workpaper, technical review or piece of advice. Grading uses criteria written by tax experts, and an attempt passes only if it satisfies at least 95% of them, covering calculations, supporting evidence, consequential issues and the completed deliverable. The 50 environments hold 1,789 workspace files, a median of 30 and as many as 123 per assignment, including 1,487 PDFs, 117 Excel workbooks and 38 Word documents alongside emails, images, CSVs and JSON records; tax experts built the tasks, and each went through a three-layer review for correctness, completeness and solvability.

Much of the weight in these assignments falls on issues the client never raises, and that judgment is not written into the initial instructions. In one condo-sale case both proposed closing dates met the residence test for an exclusion of up to $250,000 in gains, but that eligibility would lapse around the end of August 2027; Astra never identified the deadline across five attempts, while Opus identified it in all five. The benchmark spans eight areas of tax practice across 50 environments, with deliverables ranging from full filing packages to strategy letters, and one assignment pairs a client's assumption of operating only in Alabama with a dispatch log showing crews in five states.

The demonstrated scope is 50 tax environments across eight practice areas, scored as pass@1 against a 95% rubric threshold. The full leaderboard sits on the DAYJOB: Tax benchmark page, and Surge AI invites requests for access to the full benchmark at benchmarks@surgehq.ai, aimed at researchers and engineers working on agents for professional workflows.

Surge AI reads the results as showing that even when agents can perform the individual steps, exercising professional judgment and reliably carrying out real-world tax workflows remain out of reach for frontier models. The assignments deliberately leave instructions incomplete, records contain conflicting evidence, and agents must decide for themselves when information is insufficient.