Skip to main content
Back to timeline
arXivSource publication:

A backdoor that corrupts LLM reasoning in four stages: eight benign words, and the answer stays right while the logic breaks

Lead

A new backdoor makes a compromised model degrade its reasoning in four ordered stages as benign trigger words accumulate, going wrong before the final answer does, and existing defenses do not fully break it.

Source-provided article image: Cancerous Backdoor Attack: Gradual Activation via Invisible Tokens and A Stage Controller
Figure 1 · arXiv

Story

A backdoored model no longer turns malicious at a single trigger point; it passes through four ordered behavior stages as trigger exposure accumulates, with reasoning failing before the answer does. Prior backdoor attacks largely follow a single-point activation paradigm, where satisfying a trigger condition produces one terminal malicious behavior; benign-trigger, multi-trigger, temporal and reasoning-path work changes how the condition is built, when it is reached, or where the effect appears, but the malicious behavior still centers on one activated target. Across three reasoning datasets (EntailmentBank, PrOntoQA, GSM8K) and three instruction-tuned models (Llama, Qwen, Mistral), the intended behavior is the largest observed reasoning-answer outcome in all 36 model-dataset-stage settings; mean attack success at intermediate stages S2/S3 is 75.56%/77.78% on EB, 95.24%/93.33% on PQA, and 95.56%/93.33% on G8K.

The attack separates visible benign trigger words from hidden stage markers the user never sees, and an inference-time stage controller picks the hidden marker from the accumulated trigger count so one compromised model walks the ordered stages. Conventional backdoors bind the trigger condition directly to a single malicious target, whereas here the identity of a visible trigger does not determine the stage, and the same trigger appears across all four stages of training data to avoid a direct trigger-to-stage association. Training extends the tokenizer with four hidden stage-conditioning tokens and resizes the embedding matrix, then jointly fine-tunes a single model on 1,800 examples with QLoRA and LoRA adapters; at inference the controller only maintains a cumulative unique trigger count and selects a fixed hidden token, without modifying generated responses or constructing stage behavior itself.

Removing an intermediate stage or the hidden stage markers collapses the intended gradual activation, showing intermediate behaviors do not emerge automatically from neighboring stage supervision. If intermediate stages were a byproduct of the terminal behavior, they should survive removal of neighboring stage supervision, but ablation shows a selective collapse of the omitted stage. Removing S2 drops its attack success from 83.33% to 3.33% (Llama), 70.00% to 6.67% (Qwen), and 73.33% to 13.33% (Mistral); removing the hidden stage tokens drops mean S4 success on EntailmentBank to 1.11%, with per-model S4 of 0.00%, 3.33%, and 0.00%.

What to watch

The work is conducted in controlled research environments using publicly available model families and benchmark reasoning datasets, and the threat model explicitly assumes adversarial access to the model's post-training pipeline and input-processing path rather than demonstrating unauthorized compromise of a deployed service. It targets security and model-development readers studying backdoor activation structure and defense evaluation, with evaluation spanning reasoning tasks including EntailmentBank, PrOntoQA, GSM8K, and ProofWriter.

The authors note that gradual activation refers to an ordered change in target behavior, not a monotonic increase in attack success values, and that in S2 and S3 reasoning quality degrades while the final answer stays correct. In zero-shot transfer, the mildest S2 degradation is more sensitive to the target task, and later stages transfer less well from a simpler source task to richer reasoning. Defense evaluation shows defenses affect different parts of the gradual activation, and even the two strongest defenses do not fully break S2-to-S4 activation.

Sources