Jonathan H. Chen

Stanford Hospital ran PharmaAgent, an LLM-based vancomycin dosing agent, silently for two months and compared high- and low-autonomy configurations across 403 dosing events in 316 patients. With institutional guidelines, mean absolute percentage error against the treating pharmacist's order was 21.8% for high autonomy versus 37.8% for low autonomy, and recommendations fell within 25% of the order in 73.4% versus 50.9% of events. Yet pharmacists rated 65.8% of 196 reviewed out-of-tolerance recommendations as carrying a safety concern, and neither configuration is ready to dose patients without pharmacist oversight.

Stanford Hospital ran PharmaAgent, an LLM-based vancomycin dosing agent, silently for two months and compared high- and low-autonomy configurations across 403 dosing events in 316 patients. With institutional guidelines, mean absolute percentage error against the treating pharmacist's order was 21.8% for high autonomy versus 37.8% for low autonomy, and recommendations fell within 25% of the order in 73.4% versus 50.9% of events. Yet pharmacists rated 65.8% of 196 reviewed out-of-tolerance recommendations as carrying a safety concern, and neither configuration is ready to dose patients without pharmacist oversight.