# Entropy-aware quantization

Edge LLM weight formats spend equal bits on every integer code, although the codes are far from uniform. LazyWeights keeps llama.cpp's GGUF formats, tunes GPTQ so the codes concentrate, Huffman-codes them losslessly in 32 interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, leaving GPU kernels unchanged. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format; on a Jetson Orin Nano Super, with the decoder unit, latency and energy per token fall by up to 36% and 38%.

Edge LLM weight formats spend equal bits on every integer code, although the codes are far from uniform. LazyWeights keeps llama.cpp's GGUF formats, tunes GPTQ so the codes concentrate, Huffman-codes them losslessly in 32 interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, leaving GPU kernels unchanged. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format; on a Jetson Orin Nano Super, with the decoder unit, latency and energy per token fall by up to 36% and 38%.