Conditional-memory tables stop spending bits by access frequency: gating each lookup cuts DeepSeek table memory from 23.6 to 3.04 GiB
Synopsis
Large language models store part of their factual knowledge in hash tables keyed by preceding token sequences, and low-bit quantization erases long-tail facts first, while giving more bits to frequently accessed rows does worse than choosing at random. GateRQ stores each row as a ladder of residual codes, computes each lookup's gate from a 1-bit code, and reads deeper levels only for the lookups the gate ranks highest. On DeepSeek-V4.1-Flash it cuts table memory from 23.6 to 3.04 GiB per GPU and raises vLLM throughput over FP8 tables in GPU memory by about 11% and about 31% on the two workloads, losing about 1 PopQA point at 1.50 bits per lookup.
Interpretation
Precision allocation for conditional-memory tables moves from per-row statistics to a per-lookup decision driven by the model's own gate. The standard rule came from recommender-system embedding tables: give frequently accessed rows larger dimensions or higher precision, assuming every access contributes equally to the output. On PopQA, giving extra precision to the most frequently accessed share of lookups removes less of the error than giving the same precision at random, so the frequency rule spends bits where they matter least.
Each row is stored as a ladder of residual codes that can be refined after it has been read, with level 1 kept in GPU memory to compute the gate and deeper levels fetched from host memory only when needed. The scheme, GateRQ for gate-driven residual quantization, leaves the model's gating mechanism intact and needs no retraining. Across DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, three codecs (TCQ, INT, PQ) and two read budgets, GateRQ beats selection by expected gate, at random and by access frequency in all paired PopQA comparisons, and every one of those differences is significant.
Perspective
The demonstrated scope is two models with conditional-memory layers, DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, three codecs and two read budgets of 1.50 and 2.10 bits per lookup, with accuracy evaluated on the full test sets of PopQA, TriviaQA, CMMLU and NQ-open. The native vLLM implementation covers only DeepSeek, and the throughput figures come from single runs on specific workloads.
At 1.50 bits per lookup GateRQ trails FP8 by about 1 PopQA point; at 2.10 bits the gap is 0.2 to 0.3 points with the TCQ ladder and about 0.7 points with the INT ladder, so it is not consistent across codecs. Reading 1.50 or 2.10 bits shows no statistically significant PopQA difference from the full two- or three-level ladder, which does not establish equivalence. The throughput gain comes mainly from a larger KV cache rather than fewer bits read, it is a single-run measurement on specific workloads, and native four-task accuracy remains to be cross-checked against the accuracy runs.