# Linear Depth Memory

Transformers have long passed information across depth by adding layer outputs in a residual stream, and recent alternatives mHC and AttnRes make that depth aggregation more expressive. Depth Memory recasts cross-layer flow as streams that store and retrieve layer representations, and from it comes Linear Depth Memory (LDM), which treats retrieval across depth as a regression solved exactly by recursive least squares while carrying a residual stream, a matrix memory stream and an inverse covariance stream. Pretrained on 400M Llama-like and 1.5B Qwen-like backbones with 50B and 100B FineWeb-Edu tokens, LDM reaches lower validation, WikiText-2 and LAMBADA perplexity than tuned PreLN, mHC and AttnRes, with competitive zero-shot accuracy.

Transformers have long passed information across depth by adding layer outputs in a residual stream, and recent alternatives mHC and AttnRes make that depth aggregation more expressive. Depth Memory recasts cross-layer flow as streams that store and retrieve layer representations, and from it comes Linear Depth Memory (LDM), which treats retrieval across depth as a regression solved exactly by recursive least squares while carrying a residual stream, a matrix memory stream and an inverse covariance stream. Pretrained on 400M Llama-like and 1.5B Qwen-like backbones with 50B and 100B FineWeb-Edu tokens, LDM reaches lower validation, WikiText-2 and LAMBADA perplexity than tuned PreLN, mHC and AttnRes, with competitive zero-shot accuracy.