Shanghai University of Finance and Economics

Cross-layer transcoders (CLTs) decompose a large language model's layer activations into interpretable features, but layers differ widely in how hard they are to reconstruct, so giving every layer a fixed number of active latents is a poor fit. Cross-Layer TopK, from ByteDance Seed and collaborators, ranks the latents of all layers together and keeps a fixed total of K per token, letting the model decide how many each layer gets. On Qwen3-8B-Base, at the same 360 activations per token as per-layer TopK, validation FVU falls from 27.64% to 27.07%, and pairing it with sparse decoding cuts training step time to about one third.

Cross-layer transcoders (CLTs) decompose a large language model's layer activations into interpretable features, but layers differ widely in how hard they are to reconstruct, so giving every layer a fixed number of active latents is a poor fit. Cross-Layer TopK, from ByteDance Seed and collaborators, ranks the latents of all layers together and keeps a fixed total of K per token, letting the model decide how many each layer gets. On Qwen3-8B-Base, at the same 360 activations per token as per-layer TopK, validation FVU falls from 27.64% to 27.07%, and pairing it with sparse decoding cuts training step time to about one third.

Medical CLIP has mainly aligned high-level semantic features, leaving texture, entity order and attributes poorly captured and limiting fine-grained tasks. Deco-CLIP splits both image and text into two branches: one ViT pretrained by CT-CLIP for semantics, one MAE-pretrained ViT for texture, with a text encoder that forks in its last three layers, shares keys and values, and uses distinct queries, plus threshold-adaptive sample mining and a regularized attention mechanism. On CT-RATE zero-shot tasks, multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%.

Medical CLIP has mainly aligned high-level semantic features, leaving texture, entity order and attributes poorly captured and limiting fine-grained tasks. Deco-CLIP splits both image and text into two branches: one ViT pretrained by CT-CLIP for semantics, one MAE-pretrained ViT for texture, with a text encoder that forks in its last three layers, shares keys and values, and uses distinct queries, plus threshold-adaptive sample mining and a regularized attention mechanism. On CT-RATE zero-shot tasks, multi-abnormality detection accuracy improves by 10.2% and report-to-volume retrieval recall by 16.2%.

The authors show that lower reconstruction loss on calibration data in weight-only PTQ does not guarantee better model performance and can even degrade it on the same calibration data; they propose DRQ, which minimizes worst-case reconstruction loss over a constrained set of input activation distributions by updating only the integer codes of quantized weights while keeping bit width, grouping, scales, and zero points fixed, improving perplexity and average task accuracy across six PTQ methods and both dense and MoE models without adding inference overhead.

The authors show that lower reconstruction loss on calibration data in weight-only PTQ does not guarantee better model performance and can even degrade it on the same calibration data; they propose DRQ, which minimizes worst-case reconstruction loss over a constrained set of input activation distributions by updating only the integer codes of quantized weights while keeping bit width, grouping, scales, and zero points fixed, improving perplexity and average task accuracy across six PTQ methods and both dense and MoE models without adding inference overhead.

This work studies, from a representation geometry perspective, why unlearned models rapidly recover forgotten knowledge under relearning attacks, finding that existing unlearning methods mainly optimize along dominant components while leaving minor components largely unchanged, that modifications in dominant components are easily reversed during relearning whereas minor components resist such reversal, and proposes Minor Component Unlearning (MCU), which explicitly targets minor components and is validated on three datasets to substantially improve resistance to relearning attacks.

This work studies, from a representation geometry perspective, why unlearned models rapidly recover forgotten knowledge under relearning attacks, finding that existing unlearning methods mainly optimize along dominant components while leaving minor components largely unchanged, that modifications in dominant components are easily reversed during relearning whereas minor components resist such reversal, and proposes Minor Component Unlearning (MCU), which explicitly targets minor components and is validated on three datasets to substantially improve resistance to relearning attacks.

The authors introduce CATCH, a controllable testbed for reward hacking in coding RL that deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit, while controlling the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing; experiments show CATCH produces diverse RL training trajectories with clear reward hacking, that initial models and reward difficulties jointly shape its emergence, and that a chain-of-thought monitor initially suppresses hacking but this protection erodes as the policy model learns to mislead the monitor with code comments.

The authors introduce CATCH, a controllable testbed for reward hacking in coding RL that deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit, while controlling the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing; experiments show CATCH produces diverse RL training trajectories with clear reward hacking, that initial models and reward difficulties jointly shape its emergence, and that a chain-of-thought monitor initially suppresses hacking but this protection erodes as the policy model learns to mislead the monitor with code comments.

The work introduces fluctuation-supervised pretraining (FSP), labeling each synthetic table by its average treatment effect plus its efficient influence-function fluctuation while deployment remains a frozen forward pass; the author proves an endpoint transition along the path T_λ,P=θ(P)+λP_nψ_P, where every fixed λ<1 retains label ambiguity of order (1-λ)²/n whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^-2; experiments show that across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells, validation-selected Summary FSP deploys 11.

The work introduces fluctuation-supervised pretraining (FSP), labeling each synthetic table by its average treatment effect plus its efficient influence-function fluctuation while deployment remains a frozen forward pass; the author proves an endpoint transition along the path T_λ,P=θ(P)+λP_nψ_P, where every fixed λ<1 retains label ambiguity of order (1-λ)²/n whereas full fluctuation makes the Gaussian label observable and reduces optimal finite-stratum causal label-prediction risk to order n^-2; experiments show that across 24 nonlinear continuous-covariate cells at trained context lengths, continuous-row FSP lowers checkpoint-mean macro RMSE by 7.0% versus S-learner and wins all 12 weak-overlap cells, validation-selected Summary FSP deploys 11.