# Attention Routing

Using paired enabled/bypassed executions of the final attention layer to produce signed next-token loss differences in frozen SmolLM3-3B-Base and Qwen3.5-4B-Base, the study tests whether hidden-state extrapolation error, curvature and error change improve attention routing beyond uncertainty, one-step displacement, position and state projections; on 100 held-out PG-19 books at an identical causal 20% invocation quota, none of six prespecified comparisons shows a positive gain after Holm correction, and in Qwen3.5 a parameter-matched fixed-projection control lowers NLL by 0.00356 nats/token relative to the trajectory router.

Using paired enabled/bypassed executions of the final attention layer to produce signed next-token loss differences in frozen SmolLM3-3B-Base and Qwen3.5-4B-Base, the study tests whether hidden-state extrapolation error, curvature and error change improve attention routing beyond uncertainty, one-step displacement, position and state projections; on 100 held-out PG-19 books at an identical causal 20% invocation quota, none of six prespecified comparisons shows a positive gain after Holm correction, and in Qwen3.5 a parameter-matched fixed-projection control lowers NLL by 0.00356 nats/token relative to the trajectory router.

The study runs three connected analyses (alignment decoupling, attention routing and entropy, intrinsic dimensionality) plus causal intervention on representative concatenation and native multimodal LLMs, finding that concatenation models follow a text-first, vision-later fusion pathway while native models show earlier visual-textual co-adaptation and feature-space reorganization, offering a mechanistic view of multimodal fusion and architecture-aware diagnostics.

The study runs three connected analyses (alignment decoupling, attention routing and entropy, intrinsic dimensionality) plus causal intervention on representative concatenation and native multimodal LLMs, finding that concatenation models follow a text-first, vision-later fusion pathway while native models show earlier visual-textual co-adaptation and feature-space reorganization, offering a mechanistic view of multimodal fusion and architecture-aware diagnostics.

Using activation patching across twenty-five models spanning eight LLM families, the work identifies an early-layer (L0) attention routing circuit shared by VQ-tokenized vision-language models, proposes a three-gate diagnostic that isolates ten models carrying it and rejects the other fifteen, shows via a single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) that vector quantization is the source of the pathological signal, and finds that only L0 ablation reduces object hallucination in open-ended generation (CHAIR_i down 31% relative) while tuned DoLA and VCD do not.

Using activation patching across twenty-five models spanning eight LLM families, the work identifies an early-layer (L0) attention routing circuit shared by VQ-tokenized vision-language models, proposes a three-gate diagnostic that isolates ten models carrying it and rejects the other fifteen, shows via a single-variable architectural swap (LLaVA-1.6 CLIP+MLP to VQ+Linear) that vector quantization is the source of the pathological signal, and finds that only L0 ablation reduces object hallucination in open-ended generation (CHAIR_i down 31% relative) while tuned DoLA and VCD do not.