Medical AI aces licensing exams but finishes under half the chart work: two execution-grounded benchmarks point the same way

Medical AI evaluation now targets continuous multi-step execution inside electronic health records and administrative portals, measuring what actually changed in the environment. Earlier evaluation relied on licensing exams and single-turn prompt scoring, which show whether a model can produce the right answer but not whether it can open a chart, reconcile an old study, place orders and write the note. Two independently built, execution-grounded benchmarks, PhysicianBench for clinical work and HealthAdminBench for administrative work, both find today's best agents finishing less than half the work; the defensible claim is directional rather than a direct comparison of the two numbers.

PhysicianBench rewrites real consultations as long-horizon tasks averaging 27 tool calls, run against synthetic records in an EHR sandbox, with scripts checking whether orders truly entered the record. Single-turn scoring only reads the text a model produces, while these tasks hinge on whether step seven happens, so task-specific scripts inspect the orders in the record and model judgments cover reasoning and notes. The 100 tasks are adapted from real primary care and subspecialist consultations across 21 specialties, independently reviewed by a physician panel, with 670 checks in total; across 12 models the best, GPT-5.5, completed 46% on a single attempt and only 28% in all three runs.

HealthAdminBench moves billing, authorization and appeals into four simulated web environments that agents operate through screenshots, clicks and typing, as a person would. Administration and workflow was the weakest category in MedHELM, and this work previously had no executable environment for evaluation. More than 100 hours observing administrators at Stanford Health Care informed an EHR, two payer portals and a fax system; 135 tasks carry 1,698 verifiers, and the best agent, Claude Opus 4.6, completed 36.3% end-to-end while GPT-5.4 passed 82.8% of subtasks but only 26.7% of whole tasks.

The benchmarks address people deciding on medical AI adoption and agent developers, covering clinical decision support, clinical note generation, patient communication and education, medical research assistance, and administration and workflow. PhysicianBench runs against synthetically reconstructed records in an EHR sandbox, and HealthAdminBench runs in simulated EHR, payer portal and fax environments, all with synthetic patient data and released environments and leaderboards. Completion rates are not directly comparable across the two because environments, model sets, interfaces and grading differ; only the directional claim is supported. MedHELM is community-owned under an Apache 2.0 license, and the team invites contributions of workflow-level benchmarks.

Completion rates come from each benchmark's own model set and run counts, and GPT-5.5's 28% consistency across three runs shows single-attempt scores fluctuate. About half of failed checkpoints trace to incomplete clinical reasoning, mostly missing links and near misses rather than outright wrong conclusions. Administrative agents rarely return to fix an earlier step and run out of budget. Treating agent architecture and memory as the fix is the authors' directional judgment, not a verified result.