Ning Lü

This study had eight large language models (ChatGPT-5.5 Instant, Claude Sonnet 4.6, Gemini 3 Flash, Grok 4.20, Microsoft Copilot, Doubao Seed 1.8, DeepSeek-V4, and Qwen 3.6) answer ten patient-oriented pancreatic cancer questions on three consecutive days, with five evaluators blinded to platform identity rating all 240 responses across six dimensions of clinical quality (1,200 ratings), finding that overall performance differed among platforms, with Grok 4.20 and DeepSeek-V4 having the highest mean scores and Doubao Seed 1.8 the lowest, differences being greatest in actionability and specificity, adjacent-day score fluctuation being most pronounced for Doubao Seed 1.8 and Qwen 3.

This study had eight large language models (ChatGPT-5.5 Instant, Claude Sonnet 4.6, Gemini 3 Flash, Grok 4.20, Microsoft Copilot, Doubao Seed 1.8, DeepSeek-V4, and Qwen 3.6) answer ten patient-oriented pancreatic cancer questions on three consecutive days, with five evaluators blinded to platform identity rating all 240 responses across six dimensions of clinical quality (1,200 ratings), finding that overall performance differed among platforms, with Grok 4.20 and DeepSeek-V4 having the highest mean scores and Doubao Seed 1.8 the lowest, differences being greatest in actionability and specificity, adjacent-day score fluctuation being most pronounced for Doubao Seed 1.8 and Qwen 3.