Finance agents meet an open world: none of 150 sessions passes, and all 35 the simulated user approved fail a program check

Financial agent evaluation no longer fixes the task, the data, or a clean timeline; the agent must deliver verifiable research results while intent, data and time all stay open. Earlier financial benchmarks close the world on at least one of intent, data or timeline, which removes silent failures instead of measuring them. Across 150 sessions with four coding agents and a tool-free model, no session is accepted, and pre-veto quality ranges only from 20.2 to 37.2 out of 100.

Grading needs no reference answer: programs recompute the delivery, requiring traceable data, no dated row after the cutoff, and reported returns and drawdowns that can be rebuilt from the delivered trade log. A simulated user holds a hidden strategy sampled from a compositional grammar and reveals each detail only when asked, so eliciting the request becomes part of what is tested. Each session is graded on 45 dimensions, 15 of them vetoes; all 35 sessions the user signed off fail at least one program check, 29 deliver no trade log or prices, and 15 use data after the cutoff or fill trades before their signal.

The evaluation covers ten frozen tasks with three independent sessions per agent, and its grammar covers long-only US equity strategies with a fixed-cost model; the simulator, the LLM judge and one evaluated agent come from the same model family, a risk that program checks and expert annotation limit but do not remove.

Most agent turns hit the 300-second limit, so longer turns may let agents get further; program checks cannot prove every figure is real, and feedback scores can leak information about the hidden strategy across attempts, so leaderboard results should come from frozen agents.