Self-consistency backfires on long context: seven of eight samples agree on the same wrong answer

Self-consistency stops helping on long-context tasks and generally degrades performance, the opposite of its behavior on short content. Self-consistency had been treated as a general strategy that improves large language model reliability across tasks and domains, with its effectiveness established mainly on short content. Across 56 dataset-model pairs, only three show statistically significant improvement, while ROUGE declines significantly in 79% of pairs, for example a 5.0 drop for Qwen-2.5-3B on GovReport.

The decline comes from positional bias: models systematically over-rely on certain context regions, and self-consistency amplifies that bias instead of canceling it. Self-consistency assumes errors across samples are independent, which is what lets aggregation filter out random mistakes. On NQ-Open question answering, accuracy follows a U-shaped curve across gold document positions and drops in the middle, while text retrieval declines monotonically, with accuracy falling about 40% when the gold document sits in the final third of a 30-document context.

Errors are already highly correlated before aggregation, so the eight samples often fail together. If errors were independent, a meaningful share of the eight samples should land on the correct answer. In 30-document retrieval contexts with the answer at the end, nearly 60% of cases have none of the eight intermediate samples identify the source document, and one question-answering example has seven of eight generations give the same wrong answer.

The evaluation covers contexts up to about 30 documents, or 5 to 10K tokens, on text tasks spanning summarization, question answering, multi-hop reasoning, and controlled position probing, and it addresses researchers and practitioners who use sampling-based aggregation on long inputs. The authors position the work as problem characterization rather than solution development, and the discussion proposes testable follow-ups such as position-aware aggregation, contrastive sampling, attention recalibration, and combining retrieval augmentation with self-consistency.

The authors note that evaluation covers contexts up to about 30 documents, leaving open whether different dynamics emerge at 100K tokens and beyond; the work focuses on text tasks, so multimodal long-context settings remain unclear; and while the controlled NQ-Open setup isolates positional bias, real-world contexts may show more complex positional interactions.