Skip to main content

# GGUF

2 related articles

Subscribe to updates

JustVugg/colibri Oct 31, 2026
colibri runs hundred-billion-parameter MoE models on ordinary PCs: experts stay on disk, and 8 GB of RAM is enough to start
JustVugg released colibri, an open-source inference engine written in pure C with one file per model family and no GPU required. A mixture-of-experts model uses only a small part of itself for each token, so colibri keeps that part in RAM and reads the remaining experts from disk on demand. In the official example, the 744B-parameter int4 GLM-5.2 MoE model is ready in 32 seconds on CPU with 9.9 GB resident; the minimum is 8 GB of RAM, and the smallest model needs 22 GB of disk.
JustVugg released colibri, an open-source inference engine written in pure C with one file per model family and no GPU required. A mixture-of-experts model uses only a small part of itself for each token, so colibri keeps that part in RAM and reads the remaining experts from disk on demand. In the official example, the 744B-parameter int4 GLM-5.2 MoE model is ready in 32 seconds on CPU with 9.9 GB resident; the minimum is 8 GB of RAM, and the smallest model needs 22 GB of disk.
colibri Mixture-of-Experts GGUF
Hugging Face Sep 22, 2026
Transformers now runs llama.cpp quants
This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.
This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.
Hugging Face
GGUF Metal kernels Local inference on Apple Silicon transformers

You’re all caught up