Skip to main content

# Attention Residuals

2 related articles

Subscribe to updates

arXiv Oct 8, 2026
Retro makes tabular foundation models refine predictions earlier and more broadly, reaching top-three overall Elo on TabArena, TALENT, and RelArena
Tracing per-query predictive changes through TabICLv2, TabPFN-3, and EXAONE-Tabular, the authors find refinement is highly uneven across depth and often concentrated in later layers; they propose retrospective inference and build Retro, which uses Attention Residuals to adaptively reweight contributions from different depths and query-conditioned gated attention to modulate contextual updates element-wise, shifting refinement earlier and more broadly and reaching top-three overall Elo on TabArena, TALENT, and RelArena while lying on the performance-cost Pareto frontier.
Tracing per-query predictive changes through TabICLv2, TabPFN-3, and EXAONE-Tabular, the authors find refinement is highly uneven across depth and often concentrated in later layers; they propose retrospective inference and build Retro, which uses Attention Residuals to adaptively reweight contributions from different depths and query-conditioned gated attention to modulate contextual updates element-wise, shifting refinement earlier and more broadly and reaching top-three overall Elo on TabArena, TALENT, and RelArena while lying on the performance-cost Pareto frontier.
Nanjing University
Hao-Run Cai（First author）
Retro Retrospective Inference Attention Residuals Gated Attention Tabular Foundation Models
arXiv Sep 29, 2026
DepthBench sweeps width–depth ratio across 10 architectures at fixed parameter budget: HC and Full AttnRes keep lowering validation loss at extreme deep-narrow shapes while Pre-LN degrades
The work introduces DepthBench, a controlled benchmark that varies the width–depth aspect ratio (d_model/n_layer) from shallow-wide to deep-narrow under a fixed parameter budget and pre-training recipe, comparing 10 representative residual and normalization architectures, and finds that the benefit of depth is strongly architecture-dependent: Pre-LN and most of its norm- and scaling-based variants worsen as models get deeper and narrower (Pre-LN rises from 2.759 to 2.782), whereas HC and Full AttnRes improve consistently (HC from 2.729 to 2.699, Full AttnRes from 2.751 to 2.718), accompanied by stronger cross-layer causal dependence and order sensitivity, though deeper models also incur compute and memory overhead.
The work introduces DepthBench, a controlled benchmark that varies the width–depth aspect ratio (d_model/n_layer) from shallow-wide to deep-narrow under a fixed parameter budget and pre-training recipe, comparing 10 representative residual and normalization architectures, and finds that the benefit of depth is strongly architecture-dependent: Pre-LN and most of its norm- and scaling-based variants worsen as models get deeper and narrower (Pre-LN rises from 2.759 to 2.782), whereas HC and Full AttnRes improve consistently (HC from 2.729 to 2.699, Full AttnRes from 2.751 to 2.718), accompanied by stronger cross-layer causal dependence and order sensitivity, though deeper models also incur compute and memory overhead.
Chinese University of Hong Kong
Keyu Wang（First author）
DepthBench Computational Depth Residual Connection Hyper-Connections Attention Residuals

You’re all caught up