ByteDance Seed

Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.

Researchers at ByteDance Seed and Shanghai University of Finance and Economics trained a sparse autoencoder (SAE, a dictionary that reconstructs a layer's output as sparse interpretable features) on the MoE output of layers 4, 30 and 46 of Qwen3-30B-A3B, and used the encoder's affinity in the expert outputs to decompose each feature's activation exactly over the experts. Averaged over a corpus, each expert gets the set of features it principally drives; most of those features have only one or two dominators, and the division of labor runs below coarse domains, down to Java versus front-end web code and geometry versus arithmetic and polynomials.

The work introduces Reward-Weighted Transport Distillation (RWTD), a post-training method for one-step generators that needs only generated samples and scalar reward evaluations, builds an adaptive target mixing separately reward-tilted current and reference distributions, realizes it through feature-space optimal transport and fixed-point regression, and raises the one-step SANA Sprint 1.6B GenEval score from 0.73 to 0.80 while showing cross-reward generalization in preference alignment experiments.

The work introduces Reward-Weighted Transport Distillation (RWTD), a post-training method for one-step generators that needs only generated samples and scalar reward evaluations, builds an adaptive target mixing separately reward-tilted current and reference distributions, realizes it through feature-space optimal transport and fixed-point regression, and raises the one-step SANA Sprint 1.6B GenEval score from 0.73 to 0.80 while showing cross-reward generalization in preference alignment experiments.

The work names and systematically characterizes phase sensitivity in models with chunked KV-cache compression: the same information is easier or harder to retrieve depending on its phase relative to compression-window boundaries, with long-context retrieval accuracy in the DeepSeek-V4 family differing by up to 40.2 percentage points across phases at 128K context; the authors reproduce the effect by pretraining 23 compressed models from scratch (Qwen3-0.6B backbone, 100B tokens), use KV-head and layer mean-replacement knockouts to reveal phase specialization, and prove in an idealized induction model that gradient flow drives compression gates toward static preferences, showing that average benchmark scores can conceal periodic positional failures.

The work names and systematically characterizes phase sensitivity in models with chunked KV-cache compression: the same information is easier or harder to retrieve depending on its phase relative to compression-window boundaries, with long-context retrieval accuracy in the DeepSeek-V4 family differing by up to 40.2 percentage points across phases at 128K context; the authors reproduce the effect by pretraining 23 compressed models from scratch (Qwen3-0.6B backbone, 100B tokens), use KV-head and layer mean-replacement knockouts to reveal phase specialization, and prove in an idealized induction model that gradient flow drives compression gates toward static preferences, showing that average benchmark scores can conceal periodic positional failures.

This work studies the training stability of full-pipeline FP8 reinforcement learning for LLMs, identifies that compounded FP8 quantization noise distorts the importance ratio so that negative-advantage tokens are over-clipped at the lower bound and their gradients are erroneously zeroed, and proposes Calibrated Clipping, which dynamically realigns clipping bounds by matching the lower-bound clipping quantile of a BF16 reference and rebalancing the upper bound, eliminating entropy surges and restoring near-BF16 performance across GRPO and DAPO, 8B to 32B models, and multiple FP8 scaling granularities, while reaching up to 1.5x the BF16 training throughput.

This work studies the training stability of full-pipeline FP8 reinforcement learning for LLMs, identifies that compounded FP8 quantization noise distorts the importance ratio so that negative-advantage tokens are over-clipped at the lower bound and their gradients are erroneously zeroed, and proposes Calibrated Clipping, which dynamically realigns clipping bounds by matching the lower-bound clipping quantile of a BF16 reference and rebalancing the upper bound, eliminating entropy surges and restoring near-BF16 performance across GRPO and DAPO, 8B to 32B models, and multiple FP8 scaling granularities, while reaching up to 1.5x the BF16 training throughput.

Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.

Researchers from Shanghai Jiao Tong University and ByteDance Seed propose PISA, a block-sparse attention mechanism that uses coarse-to-fine pyramid Top-K selection with LogSumExp scoring and hardware-aware Triton kernels to reduce block-selection complexity from O(N²/C) to O(N log N); across 418M, 1.47B, and 2.67B scales it matches BSA, NSA, and HiLS on language modeling and commonsense reasoning, achieves the highest average accuracy among sparse methods on six containment tasks, and speeds up block selection over BSA by 2.86x, 5.31x, and 9.95x at 64K, 128K, and 256K.