# Sparse Decoding

Cross-layer transcoders (CLTs) decompose a large language model's layer activations into interpretable features, but layers differ widely in how hard they are to reconstruct, so giving every layer a fixed number of active latents is a poor fit. Cross-Layer TopK, from ByteDance Seed and collaborators, ranks the latents of all layers together and keeps a fixed total of K per token, letting the model decide how many each layer gets. On Qwen3-8B-Base, at the same 360 activations per token as per-layer TopK, validation FVU falls from 27.64% to 27.07%, and pairing it with sparse decoding cuts training step time to about one third.

Cross-layer transcoders (CLTs) decompose a large language model's layer activations into interpretable features, but layers differ widely in how hard they are to reconstruct, so giving every layer a fixed number of active latents is a poor fit. Cross-Layer TopK, from ByteDance Seed and collaborators, ranks the latents of all layers together and keeps a fixed total of K per token, letting the model decide how many each layer gets. On Qwen3-8B-Base, at the same 360 activations per token as per-layer TopK, validation FVU falls from 27.64% to 27.07%, and pairing it with sparse decoding cuts training step time to about one third.