DictLLM shrinks a 7B model 5.1× with shared dictionaries and sparse coefficients, adding 2.9 points of perplexity
Synopsis
On mobile devices the binding constraint for large language models is weight memory, not computation. DictLLM splits weight matrices within grouped layers into one shared dictionary plus layer-specific sparse coefficients, all computed from a pretrained model without retraining. On LLaMA-7B it compresses the model 5.1× with only a 2.9-point rise in WikiText-2 perplexity, a compression ratio about 3.6× that of the state-of-the-art low-rank method.
Interpretation
DictLLM represents the weight matrices of a group of layers as one shared dictionary plus each layer's own sparse coefficients, so the compressed model stores a single dictionary per group along with sparse coefficients. Earlier low-rank compression truncates each weight matrix separately, and performance degrades sharply when the rank is pushed low; dictionary-based parameter sharing usually trains the shared representation from scratch, which is hard to apply to billion-scale pretrained models. On LLaMA-7B, DictLLM compresses the model 5.1× with only a 2.9-point increase in WikiText-2 perplexity, about 3.6× the compression ratio of the state-of-the-art low-rank method; across compression ratios from 1.25× to 2× it beats ASVD, SVD-LLM and Basis Sharing on WikiText-2, PTB and C4 perplexity and on average accuracy over seven zero-shot reasoning benchmarks.
Compression runs in four steps: singular value decomposition of input-scaled weights yields the shared dictionary, coefficients are pruned in the dictionary-induced coefficient space, the dictionary is refitted to the sparse coefficients, and both factors are quantized to 4-bit floating point. A dictionary-aware Hessian projects the calibration input covariance into coefficient space, so pruning and quantization act on the coefficients rather than on dense weights. With both dictionary and coefficients quantized to NVFP4, LLaMA-7B reaches average zero-shot accuracies of 0.47, 0.43 and 0.40 at 5×, 6× and 7× compression; the 0.40 at 7× still exceeds the 0.36 that Basis Sharing reaches at 2×.
Perspective
The demonstrated scope is post-training compression of LLaMA-7B, LLaMA2-7B, OPT-6.7B, Mistral-7B and Qwen3-1.7B/8B/14B, with calibration from 256 WikiText-2 samples and evaluation on WikiText-2, PTB and C4 perplexity plus seven zero-shot reasoning benchmarks. End-to-end decoding throughput and memory were measured only on a single NVIDIA RTX PRO 6000 Blackwell Max-Q at batch size 1 with the MACKO sparse backend.
Coefficient pruning produces unstructured sparsity, so hardware efficiency depends on support for irregular sparse access patterns, and structured sparsity is named as a direction for further work. Because the compressed model stays close to the original pretrained model, it may also retain that model's biases, safety risks and failure modes.