Vancomycin dosing agents ran silently for two months at Stanford: the high-autonomy setup cut error to 21.8%, yet pharmacists still had to review every dose
Synopsis
Stanford Hospital ran PharmaAgent, an LLM-based vancomycin dosing agent, silently for two months and compared high- and low-autonomy configurations across 403 dosing events in 316 patients. With institutional guidelines, mean absolute percentage error against the treating pharmacist's order was 21.8% for high autonomy versus 37.8% for low autonomy, and recommendations fell within 25% of the order in 73.4% versus 50.9% of events. Yet pharmacists rated 65.8% of 196 reviewed out-of-tolerance recommendations as carrying a safety concern, and neither configuration is ready to dose patients without pharmacist oversight.
Interpretation
Vancomycin dosing, a multi-turn task health systems already delegate from physicians to pharmacists under protocol, now has a real-record comparison of LLM agents: the same underlying model gave recommendations either through a developer-fixed workflow (low autonomy) or by choosing its own action order (high autonomy). Agentic systems had mostly been validated in simulated clinical environments, while real deployments largely kept stronger human control through more constrained, highly specified workflows; how much runtime autonomy to grant was unclear. In a silent deployment at Stanford Hospital from July 28 to October 1, 2026, 403 dosing events in 316 patients received recommendations from all four configurations without influencing care; with guidelines, mean absolute percentage error was 21.8% for high autonomy versus 37.8% for low autonomy, a paired mean difference of −16.01 percentage points (95% CI, −19.25 to −12.77).
The high-autonomy configuration let the model write and execute Python code to query FHIR interfaces and the pharmacokinetic engine, see errors inside its loop, and retry until it finalized a dose. The low-autonomy configuration fixed the order of data retrieval, acute kidney injury assessment and InsightRx engine calls in developer code, leaving the model bounded tasks within that sequence and unable to recover from transient interface errors. Both configurations shared one model, one Bayesian pharmacokinetic engine and one output format, and each ran with and without institutional guidelines; among 196 pharmacist-reviewed out-of-tolerance recommendations, low autonomy carried higher odds of a safety concern (odds ratio 2.37; 95% CI, 1.30 to 4.37).
Supplying the institutional dosing guideline in the prompt produced no statistically significant difference in agreement with pharmacist orders. In paired comparisons, the mean absolute percentage error difference between guideline and no-guideline conditions was −0.93 percentage points for high autonomy and −1.95 for low autonomy, neither significant after correction. The share within the 25% tolerance was 73.4% versus 73.9% for high autonomy and 50.9% versus 50.9% for low autonomy, with McNemar P = 1.00 in both cases.
Perspective
The results come from a prospective silent deployment at a single center, Stanford Health Care, for one drug, vancomycin, and one model; recommendations were generated and stored in a research environment, neither shown to clinicians nor written to the EHR, so the study measures agreement with pharmacist orders and pharmacist-rated safety concerns rather than drug exposure or patient outcomes. Only patient-days with a single scheduled order and no dialysis were analyzed, and the 403 events were treated as independent.
Total daily dose is a coarse metric that treats regimens with different intervals as equivalent; configurations ran after the day's order was written, and same-day orders were filtered while later laboratory results remained visible; neither configuration could read medication administration records or vancomycin level timestamps, which made dosing history and level interpretation the most common error category. The authors state plainly that neither configuration is ready to dose patients autonomously.