Medical AI aces licensing exams but finishes under half the chart work: two execution-grounded benchmarks point the same way
Medical AI evaluation is shifting from answering questions to acting inside real work environments. Near-perfect licensing exam scores said nothing about whether a model can carry a task across records, documents and software systems. Using an EHR sandbox and simulated payer portals, the best agents completed 46% and 36.3% of clinical and administrative tasks on a single attempt, mostly losing information between steps.