# Memory-efficient LLM inference

Edge LLM weight formats spend equal bits on every integer code, although the codes are far from uniform. LazyWeights keeps llama.cpp's GGUF formats, tunes GPTQ so the codes concentrate, Huffman-codes them losslessly in 32 interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, leaving GPU kernels unchanged. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format; on a Jetson Orin Nano Super, with the decoder unit, latency and energy per token fall by up to 36% and 38%.

Edge LLM weight formats spend equal bits on every integer code, although the codes are far from uniform. LazyWeights keeps llama.cpp's GGUF formats, tunes GPTQ so the codes concentrate, Huffman-codes them losslessly in 32 interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, leaving GPU kernels unchanged. At 1% and 3.5% perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are 5.7–31.4% smaller than the best llama.cpp format; on a Jetson Orin Nano Super, with the decoder unit, latency and energy per token fall by up to 36% and 38%.

BitNest introduces a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation and recovers the target through residual refinement, so draft and target share a single physical weight representation, with the progressive-precision design extended to the KV cache for long-context inference; across multiple 7B–8B edge-friendly LLMs and diverse workloads it reports an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and 1.48–1.61x end-to-end speedup over FP16 autoregressive decoding, with consistently competitive or higher decoding speedup on the LLaMA models supported by all representative self-speculative baselines.

BitNest introduces a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation and recovers the target through residual refinement, so draft and target share a single physical weight representation, with the progressive-precision design extended to the KV cache for long-context inference; across multiple 7B–8B edge-friendly LLMs and diverse workloads it reports an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and 1.48–1.61x end-to-end speedup over FP16 autoregressive decoding, with consistently competitive or higher decoding speedup on the LLaMA models supported by all representative self-speculative baselines.