# Model compression

The information a fine-tune writes into its adapter is now measured not by parameter count or weight reconstruction error but as the smallest serialized size of a compressed adapter that, reloaded into the frozen model, keeps 90% of the held-out fine-tuning gain. The same dataset demands very different rates across frozen models: keeping 90% of the math gain takes 1.03 bits per value on Qwen2.5-Math-7B but 0.41 on Qwen2.5-7B, and on an adapter trained with deliberately mismatched reasoning supervision, compression lifts GSM8K accuracy from 23.7% to 82.9%, above the frozen Qwen2.5-7B's 77.6%.

The information a fine-tune writes into its adapter is now measured not by parameter count or weight reconstruction error but as the smallest serialized size of a compressed adapter that, reloaded into the frozen model, keeps 90% of the held-out fine-tuning gain. The same dataset demands very different rates across frozen models: keeping 90% of the math gain takes 1.03 bits per value on Qwen2.5-Math-7B but 0.41 on Qwen2.5-7B, and on an adapter trained with deliberately mismatched reasoning supervision, compression lifts GSM8K accuracy from 23.7% to 82.9%, above the frozen Qwen2.5-7B's 77.6%.

LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.

LLM quantization usually gives an entire layer one bit width, ignoring that weight blocks within a layer differ in sensitivity to quantization noise. A2BQ scores each 128-row weight block by post-softmax attention variance from an FP16 model on a WikiText-2 calibration set, allocates 4/6/8 bits under a 5.7-bit average budget, and sets qLoRA+ LoRA ranks from layer sensitivity and assigned precision. On TinyLlama-1.1B, the attention-scope configuration with three protected layers at each boundary reaches 7.884 test perplexity at 73.11 tokens/s, while runtime GPU memory stays near FP16 because the reference backend dequantizes weights into floating-point buffers before computing.

Trained anchor-based Gaussian-splat scenes stay too large for phones and headsets, and existing post-hoc converters saturate early. This scheme treats a trained model as a read-only artifact, fits a per-scene hash-grid entropy context with per-channel Laplace distributions, and codes attributes into a Morton-ordered, progressively decodable bitstream. Across 13 scenes on three benchmarks, the post-hoc mode averages 6.0x compression at a -0.03 dB quality change and runs unchanged on third-party checkpoints.

Trained anchor-based Gaussian-splat scenes stay too large for phones and headsets, and existing post-hoc converters saturate early. This scheme treats a trained model as a read-only artifact, fits a per-scene hash-grid entropy context with per-channel Laplace distributions, and codes attributes into a Morton-ordered, progressively decodable bitstream. Across 13 scenes on three benchmarks, the post-hoc mode averages 6.0x compression at a -0.03 dB quality change and runs unchanged on third-party checkpoints.

Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.

Mixture-of-experts weight precision carries two separate costs, one for storage and one for decode; DuoRate stores a single 4.27 bpw copy and picks a read prefix per workload from that workload's own prefill, reaching 76.16 average accuracy at a 2.9 bpw read rate on Qwen3-30B-A3B, close to 4-bit formats.

Low-rank compression previously assigned a fixed rank during inference, so every token cost the same compute. LRCC adds one lightweight router per Transformer block that selects among a few nested low-rank paths for each token, training only the routers while the low-rank factors stay frozen. Within the same average active-parameter budget, Llama-2-7B gains 7.6 percentage points in average downstream accuracy over static low-rank compression; at matched batch-size-1 decoding latency, Llama-3.2-1B improves both perplexity and downstream accuracy and Llama-2-7B stays competitive, without specialized CUDA kernels.

Low-rank compression previously assigned a fixed rank during inference, so every token cost the same compute. LRCC adds one lightweight router per Transformer block that selects among a few nested low-rank paths for each token, training only the routers while the low-rank factors stay frozen. Within the same average active-parameter budget, Llama-2-7B gains 7.6 percentage points in average downstream accuracy over static low-rank compression; at matched batch-size-1 decoding latency, Llama-3.2-1B improves both perplexity and downstream accuracy and Llama-2-7B stays competitive, without specialized CUDA kernels.

The work identifies the BOS sink phenomenon as a key mechanism driving layer-wise redundancy sensitivity in large language models, showing that attention heads with high BOS sink scores, especially in deeper layers, contribute little to predictive performance and effectively serve as dumping grounds for superfluous attention weights, and introduces a simple pruning strategy that removes high-BOS sink heads; experiments on Gemma-3, Llama-3.1, and Qwen3 show this approach identifies redundant transformer components more reliably than weight- and activation-based criteria in terms of downstream task retention, remaining close to dense baselines at low-to-moderate pruning ratios, and that high-scoring sink heads sustain their focus on BOS as context length grows.

The work identifies the BOS sink phenomenon as a key mechanism driving layer-wise redundancy sensitivity in large language models, showing that attention heads with high BOS sink scores, especially in deeper layers, contribute little to predictive performance and effectively serve as dumping grounds for superfluous attention weights, and introduces a simple pruning strategy that removes high-BOS sink heads; experiments on Gemma-3, Llama-3.1, and Qwen3 show this approach identifies redundant transformer components more reliably than weight- and activation-based criteria in terms of downstream task retention, remaining close to dense baselines at low-to-moderate pruning ratios, and that high-scoring sink heads sustain their focus on BOS as context length grows.

The work proposes Displacement-Preserving Relational Distillation (DPRD), which in nnU-Net aligns batch-wise pairwise displacement vectors on ROI-masked case-level embeddings with relational scale normalization, and on ISLES 2022 and AMOS 2022 lets students using about 5% of the teacher's parameters and about 3% of its FLOPs outperform Logits KD, FitNet, RKD, and CIRKD, reaching 85.46% Dice and 11.72 mm HD95 on AMOS.

The work proposes Displacement-Preserving Relational Distillation (DPRD), which in nnU-Net aligns batch-wise pairwise displacement vectors on ROI-masked case-level embeddings with relational scale normalization, and on ISLES 2022 and AMOS 2022 lets students using about 5% of the teacher's parameters and about 3% of its FLOPs outperform Logits KD, FitNet, RKD, and CIRKD, reaching 85.46% Dice and 11.72 mm HD95 on AMOS.

The authors propose Detail Consistent Distillation (DCD), which applies a 3D discrete wavelet transform to teacher and student encoder features at every stage during training and aligns only the directional detail subband D (excluding the low-frequency approximation A and the most noise-prone extreme high-frequency band S) after inverse-wavelet reconstruction in the spatial domain, raising the mDice of a 4x channel-reduced student from 63.60% to 68.51% on BraTS 2024 and from 70.21% to 73.95% on ISLES 2022 with no inference-time overhead.

The authors propose Detail Consistent Distillation (DCD), which applies a 3D discrete wavelet transform to teacher and student encoder features at every stage during training and aligns only the directional detail subband D (excluding the low-frequency approximation A and the most noise-prone extreme high-frequency band S) after inverse-wavelet reconstruction in the spatial domain, raising the mDice of a 4x channel-reduced student from 63.60% to 68.51% on BraTS 2024 and from 70.21% to 73.95% on ISLES 2022 with no inference-time overhead.