Candidate-independent block-causal attention makes decision models insensitive to candidate ordering while retaining decision quality
Synopsis
The work introduces candidate-independent block-causal attention, which preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions; across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones, compared with standard causal attention and complementary invariant baselines, this architecture consistently reduces permutation sensitivity while retaining competitive decision quality, with ablations indicating that candidate isolation is the primary source of the effect and position resetting completing the intended symmetry, and a further Qwen3-4B study examines the architecture's behavior with substantially more training data.
Fig. 1: Structural attention masks for one prompt with four candidate blocks. Dark cells are permitted attention. Causal attention exposes earlier candidate blocks; CI attention exposes only the shared prefix and the current candidate block.
arXivInterpretation
Introduces candidate-independent block-causal attention so that a single candidate's score no longer depends on its position in the sequence. Standard causal cross-encoding is expressive but can make a candidate's score depend on serialization order rather than on the underlying decision problem; this architecture preserves causal computation within the shared context and each candidate while blocking cross-candidate information flow and resetting candidate positions. Compared against standard causal attention and complementary invariant baselines across Gemma 3 1B, Qwen3 1.7B, and Qwen3 4B backbones, reporting consistently reduced permutation sensitivity with competitive decision quality retained.
Ablations indicate candidate isolation is the primary source of the reduction in permutation sensitivity, with position resetting completing the intended symmetry. Decomposes the architectural effect into candidate isolation and position resetting, indicating their relative roles. An attribution conclusion based on ablation experiments; the abstract does not give specific ablation values.
Further examines the architecture's behavior under a larger training-data regime. Beyond the three-backbone comparison, a separate Qwen3-4B study uses substantially more training data. The abstract states only that this study exists and uses more data; no specific results are reported.
Perspective
The result targets decision models that score a variable-sized candidate set encoded in a single sequence, especially System 1 components inside generative systems where candidates may be proposed in different orders across runs. It applies where causal computation must be preserved within the shared context and each candidate while candidate scores must not change with serialization order. The authors release a code repository and a Qwen3-4B model artifact, supporting reproduction and extension on similar backbones; the larger-data Qwen3-4B study offers an entry point for observing the architecture's behavior with more data.
The abstract does not give the specific measure of permutation sensitivity, the specific decision-quality metric, the training-data scale, or ablation values, so the size of the reduction and any quality cost remain to be confirmed in the body. Results of the larger-data Qwen3-4B study are not expanded in the abstract, and whether they change the direction of the conclusion is an open question. Whether blocking cross-candidate information flow limits decision tasks that require comparing candidates against one another is another direction a reader may watch.