Skip to main content

# Computational Depth

2 related articles

Subscribe to updates

arXiv Sep 29, 2026
DepthBench sweeps width–depth ratio across 10 architectures at fixed parameter budget: HC and Full AttnRes keep lowering validation loss at extreme deep-narrow shapes while Pre-LN degrades
The work introduces DepthBench, a controlled benchmark that varies the width–depth aspect ratio (d_model/n_layer) from shallow-wide to deep-narrow under a fixed parameter budget and pre-training recipe, comparing 10 representative residual and normalization architectures, and finds that the benefit of depth is strongly architecture-dependent: Pre-LN and most of its norm- and scaling-based variants worsen as models get deeper and narrower (Pre-LN rises from 2.759 to 2.782), whereas HC and Full AttnRes improve consistently (HC from 2.729 to 2.699, Full AttnRes from 2.751 to 2.718), accompanied by stronger cross-layer causal dependence and order sensitivity, though deeper models also incur compute and memory overhead.
The work introduces DepthBench, a controlled benchmark that varies the width–depth aspect ratio (d_model/n_layer) from shallow-wide to deep-narrow under a fixed parameter budget and pre-training recipe, comparing 10 representative residual and normalization architectures, and finds that the benefit of depth is strongly architecture-dependent: Pre-LN and most of its norm- and scaling-based variants worsen as models get deeper and narrower (Pre-LN rises from 2.759 to 2.782), whereas HC and Full AttnRes improve consistently (HC from 2.729 to 2.699, Full AttnRes from 2.751 to 2.718), accompanied by stronger cross-layer causal dependence and order sensitivity, though deeper models also incur compute and memory overhead.
Chinese University of Hong Kong
Keyu Wang（First author）
DepthBench Computational Depth Residual Connection Hyper-Connections Attention Residuals
arXiv Sep 17, 2026
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Using a unified prelude–core–coda looped-transformer family, this work fits a separate compute-optimal recipe for each architecture and trains scaling ladders to about 10²⁰ FLOPs on FineWeb, finding that model growth (raising the number of core passes mid-training) and a boundary operator (normalizing the residual stream and re-injecting the prelude output) can change pre-training scaling exponents so that compute-efficiency gains widen with scale, while untying weights moves only the constant; in data-constrained multi-epoch training the optimal loop count grows with compute and scaling loops is more compute-efficient than scaling model size.
Using a unified prelude–core–coda looped-transformer family, this work fits a separate compute-optimal recipe for each architecture and trains scaling ladders to about 10²⁰ FLOPs on FineWeb, finding that model growth (raising the number of core passes mid-training) and a boundary operator (normalizing the residual stream and re-injecting the prelude output) can change pre-training scaling exponents so that compute-efficiency gains widen with scale, while untying weights moves only the constant; in data-constrained multi-epoch training the optimal loop count grows with compute and scaling loops is more compute-efficient than scaling model size.
New York University
Zixi Chen（First author）
Neural Scaling Laws Looped Transformer Model Growth Boundary Operator Computational Depth

You’re all caught up