Speech LLMs Hear Age and Gender but Misjudge When to Use It: Explicit Labels Lift Personalization and Stereotyping Together

A speech LLM's perception of a speaker's age and gender can be measured separately from whether it adapts its answer accordingly. Earlier speech evaluations either measured group disparities on content tasks or rewarded demographic-dependent changes that the user never requested, so the same behavior could count as personalization in one benchmark and stereotyping in another. BASIS-Bench splits evaluation into demographic robustness, stereotypical bias, and personalization, and compares eight speech LLMs across speech and text modes with implicit or explicit demographic labels.

Models perform comparably across demographic groups on content tasks, recognize gender reliably, and find age much harder. Content tasks cover automatic speech recognition and spoken question answering grouped by accent, gender, and age, while demographic classification asks the model to name the speaker attribute directly. Across the eight models, content-task mean accuracy ranges from 83.1 to 93.4, gender classification from 52.2 to 99.2, and age classification from 51.0 to 84.4, with AF-3 and Voxtral near zero on elderly speakers.

Supplying demographic labels in the prompt sharply improves personalization and simultaneously increases stereotypical selections. In implicit modes the model must infer demographics from speech, while explicit modes add a text label, and the gap between them separates perception failure from use failure. Gender personalization approaches ceiling for most models under speech-explicit, while stereotypical selections under explicit conditions rise by 4.7 points for female and 20.1 points for elderly speakers on average, with Qwen3-Omni showing the largest increases.

The evaluation targets speech LLMs that accept speech input, take textual instructions alongside speech, and produce textual responses, covering age and gender as the main axes while accent enters only the demographic robustness axis. Evaluation runs in four modes combining speech or text with implicit or explicit labels, which separates perception from use. It suits researchers and engineering teams concerned with behavioral consistency in speech assistants.

The evaluation covers only stereotypical bias, one form of representational bias, and not intersectional bias; gender is treated as binary and age as child, adult, and elderly, as controlled abstractions rather than full descriptions. Demographics are inferred from speech, so results measure use of perceived cues rather than speakers' self-identified identities. The personalization and stereotypical bias axes rely on synthetic voices, whose acoustic variability differs from natural speech, and the medical distinction in BASIS-Health is biological sex, which is not equivalent to gender perceived from speech.