Two-bit lookup tables shrink DeepSeek's 203 GB memory to 51 GB and cut prefill from 64.66 s to 13.27 s
Synopsis
Conditional memory lets a language model add capacity through lookup, but the whole embedding table must stay resident; DeepSeek's table alone is 203 GB. Conditional Program stores each row as a short code plus a local scale and reconstructs it with a Gaussian trellis shared across the table, decoding only the rows an input requests. At two bits, storage falls roughly four- to eightfold, most reported task scores stay close to their originals, and output KL is 41–63% lower than 2-bit round-to-nearest.
Interpretation
Lookup-based conditional memory can store reconstruction instructions instead of full embedding rows, shrinking the whole table roughly four- to eightfold. Previously such tables either stayed in accelerator memory or moved to host memory, which changes where the cost is paid without reducing it. Across DeepSeek, Qwen and LongCat, three models with hashed n-gram lookup tables, the two-bit code cuts DeepSeek's table from 202.7 GB to 50.7 GB while most reported task scores stay close to their original values.
Each row keeps only a short code and a local scale, the reconstruction rule is shared across the table, and only the rows an input requests are decoded. Conventional weight quantization targets dense computation, whereas lookup is sparse and input-dependent, and compression must still allow each row to be fetched independently. Normalized row contents are encoded with a shared Gaussian trellis and the local scale restores the original row norm, with each row stored as an independently addressable record; for a 4,096-token DeepSeek prefill from host memory, lookup takes 6.54 ms against 10.78 ms for a QTIP tile-window decoder.
Perspective
The conclusions cover post-training compression of three models with hashed lookup tables, and only the lookup tables are compressed while all other model components stay unchanged. Evaluation uses 40 tasks in five groups: commonsense, knowledge, reading comprehension, multilingual, and math and code, plus per-token output KL on six text domains. Serving measurements run on a single node with eager inference, reporting memory and the largest tested workloads on four and eight H200 GPUs. The authors note that token-indexed tables need separate treatment of sensitive rows, examined in the appendix as a boundary test.
The authors state that small task-score differences do not establish equivalence, and that some equal-budget comparisons use interpolated trellis values without uncertainty estimates. The fitted-codebook and structured-residual KL comparisons are run only on Qwen. Serving measurements use a single node and eager inference, some configurations have only one measurement run, and the tested workload limits are not exact capacity thresholds. Access counts come from English web text, so frequency-based bit allocation and caching results may change with the deployment distribution. The Gaussian source approximation does not establish how training produces the observed row statistics.