# Contrastive Decoding

Reasoning models often keep re-checking after they have reached an answer, stretching the chain of thought. DoubtLess has a small model that shares the large model's tokenizer read the same chain twice, once told to go straight to the answer and once told to double-check every step, and adds the scaled difference of the two logit vectors to the large model's logits at every step, acting only at decoding time with no weight changes. On seven competition-mathematics, graduate-science and programming benchmarks, Qwen3-32B steered by Qwen3-0.6B uses 31% fewer thinking tokens at unchanged accuracy, a larger cut than five inference-time baselines.

Reasoning models often keep re-checking after they have reached an answer, stretching the chain of thought. DoubtLess has a small model that shares the large model's tokenizer read the same chain twice, once told to go straight to the answer and once told to double-check every step, and adds the scaled difference of the two logit vectors to the large model's logits at every step, acting only at decoding time with no weight changes. On seven competition-mathematics, graduate-science and programming benchmarks, Qwen3-32B steered by Qwen3-0.6B uses 31% fewer thinking tokens at unchanged accuracy, a larger cut than five inference-time baselines.

Contrastive decoding's DoLa picks its premature layer from differences in output vocabulary distributions; Sakai and colleagues instead use self-attention distributions as the selection signal, introducing Attention-JSD, Attention-Entropy-Max and Attention-Entropy-Min while leaving the DoLa framework intact. On TruthfulQA, LLaMA-7B with Attn-Ent-Min reaches MC2 62.8% and MC3 36.5%, above DoLa's 56.5% and 30.5%; on LLaMA-33B and 65B, however, DoLa keeps the highest MC1, so the gains concentrate on multi-answer metrics.

Contrastive decoding's DoLa picks its premature layer from differences in output vocabulary distributions; Sakai and colleagues instead use self-attention distributions as the selection signal, introducing Attention-JSD, Attention-Entropy-Max and Attention-Entropy-Min while leaving the DoLa framework intact. On TruthfulQA, LLaMA-7B with Attn-Ent-Min reaches MC2 62.8% and MC3 36.5%, above DoLa's 56.5% and 30.5%; on LLaMA-33B and 65B, however, DoLa keeps the highest MC1, so the gains concentrate on multi-answer metrics.

The work introduces LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting a looped Transformer's final prediction with an earlier recurrent pass, in either logit space with one extra output pass (LoopCD-Logits) or hidden-state space with zero output overhead (LoopCD-Hidden); across four looped Transformer families it delivers consistent gains at full recurrent depth, for example raising Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33% and Huginn's HumanEval pass@1 from 22.56% to 31.71%, and these gains let it match or exceed full-depth unguided baselines at half the recurrent loops while reducing forward FLOPs by 22.5% to 48.2%.

The work introduces LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting a looped Transformer's final prediction with an earlier recurrent pass, in either logit space with one extra output pass (LoopCD-Logits) or hidden-state space with zero output overhead (LoopCD-Hidden); across four looped Transformer families it delivers consistent gains at full recurrent depth, for example raising Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33% and Huginn's HumanEval pass@1 from 22.56% to 31.71%, and these gains let it match or exceed full-depth unguided baselines at half the recurrent loops while reducing forward FLOPs by 22.5% to 48.2%.

The authors built Tri-PvP, an 8,000-sample tri-modal conflict benchmark spanning animal, emotion, environment, and music domains in which image and audio each appear in perceptual form (real photographs or recordings) or propositional form (text-card images or TTS speech) while text is always propositional, evaluated five omni-modal LLMs across four evidence-type conditions, and found that image bias dominates in 18 of 20 model-by-condition bars and that a systematic asymmetry exists: models favor perceptual evidence in vision but propositional evidence in audio, with this bias linearly decodable from early hidden layers and only partially mitigated by contrastive decoding, which introduces a residual text bias.

The authors built Tri-PvP, an 8,000-sample tri-modal conflict benchmark spanning animal, emotion, environment, and music domains in which image and audio each appear in perceptual form (real photographs or recordings) or propositional form (text-card images or TTS speech) while text is always propositional, evaluated five omni-modal LLMs across four evidence-type conditions, and found that image bias dominates in 18 of 20 model-by-condition bars and that a systematic asymmetry exists: models favor perceptual evidence in vision but propositional evidence in audio, with this bias linearly decodable from early hidden layers and only partially mitigated by contrastive decoding, which introduces a residual text bias.