Edge LLMs stop paying fixed width per weight: LazyWeights shrinks codes and cuts Orin token time by up to 36%

The fixed width of edge LLM deployment formats is broken: the same GGUF file can be smaller at equal perplexity while the GPU still reads fixed-width codes. GPTQ, AWQ and llama.cpp's K- and I-quants spend equal bits on every integer code of a matrix, although the codes are far from uniform. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format within the same budget, with Mistral-7B shrinking 31.4% at the 1% budget.

Coded weights are expanded back to fixed-width codes between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. Software decoding previously fell 15–20 short of DRAM rate on an Orin Nano Super, making a token 2.5–3.2 slower than the same codes at fixed width. A bit-exact 0.21 mm2, 1 GHz, 7 nm ASIC decoder unit cuts Orin latency and energy per token by up to 36% and 38%, or 54% and 44% with prompt-lookup speculation; the unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).

The demonstrated scope is three models (Phi-3-mini, Mistral-7B, SmolLM-1.7B), WikiText-2 perplexity budgets of 1% and 3.5%, and two edge devices, the Jetson Orin Nano Super and the Jetson Nano; the decoder unit's latency and energy are projections from measured device numbers and pre-layout synthesis, not a fabricated SoC.

The decoder unit's latency and energy are projections, and the FPGA prototype verifies bit- and cycle-exact decoding at FPGA clocks, not 1 GHz; for streamed Mistral-7B the unit's time is not reported because the fixed-width twin is no proxy for it; Llama-3, Qwen3 and long contexts were not evaluated, and the uncoded fp16 KV cache dilutes the saving at long context.