TCM diagnosis stops being text-only: ShizhenGPT feeds tongue images, pulse, smell and sound into one 32B model
Synopsis
TCM language models have been trained on under 1GB of text and handle text alone, while TCM diagnosis itself draws on looking, listening, smelling and pulse-taking. ShizhenGPT pairs more than 100GB of text from 3,256 TCM books and web sources with over 200GB of multimodal data, including 1.2M annotated images, 200+ hours of audio and pulse, smell and ECG signals, through two-stage pretraining and instruction tuning. On five 2024–2025 TCM exams ShizhenGPT-32B averages 78.1 against 84.0 for DeepSeek-R1-671B, and on a 7,204-question TCM vision benchmark it scores 63.6 against 61.4 for Gemini-3-Pro.
Interpretation
TCM-oriented models begin to take tongue images, palm images, herb appearance, cough sounds and pulse waveforms as input rather than reading text alone. Earlier open TCM models were trained on under 1GB of text and supported text only, while general models offered only partial support for vision and audio. On a TCM vision benchmark of 7,204 multiple-choice questions, ShizhenGPT-32B averages 63.6, above Gemini-3-Pro at 61.4 and GPT-5 at 58.2, with the 7B version at 60.1.
The model turns different sensor signals into one waveform representation and hands them to the language model to answer. Non-audio signals are first converted to waveforms by linear interpolation and then share the speech encoding path. All waveforms are resampled to 16kHz and turned into 128-channel mel-spectrograms with a 25ms window and 10ms hop, pooled with stride 2 and projected into the language model by a one-layer MLP, giving pulse signals 40ms per token.
On exam-style text tasks the 32B model approaches larger proprietary models but trails the largest reasoning model. It scores highest among open models of its scale and remains behind 671B DeepSeek-R1. Averaged over five 2024–2025 TCM exams, ShizhenGPT-32B scores 78.1, DeepSeek-R1-671B 84.0 and Doubao-1.5-Pro-32k 80.6; the 7B version scores 72.9, 9.4 points above its Qwen2.5-7B-Instruct backbone.
Perspective
The demonstrated scope is five 2024–2025 TCM exams, a 7,204-question TCM vision benchmark, expert pairwise comparison on 90 real-world cases, and single-modality signal classification on pulse, smell, cough, heart sound and ECG; the authors state the model is for scientific research only and must not be used in clinical or industry settings.
The authors list scarce high-quality smell and pulse data, no testing in real clinical settings, missing modalities such as touch, and synthetic supervision that was spot-checked but not verified item by item by TCM practitioners; the signal ablation shows one model retains separate capabilities but does not establish patient-level cross-modal fusion.