# Local inference on Apple Silicon

This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.

This Hugging Face post describes how transformers can now load GGUF quantized checkpoints directly through from_pretrained with a gguf_file argument: on Apple Silicon it reuses ggml Metal kernels distributed via the kernels library (packed-quantized weight matmul, fused normalization, flash attention, gated delta net, plus a home-grown topk for MoE routing) and trims generate overhead by dropping the redundant attention mask early and deferring the stopping check asynchronously; on a MacBook Pro M2 Max the authors measure generation speed close to llama.cpp across three checkpoints, report file-size tradeoffs for Q4_K_M, Q5_K_M and Q6_K, and state that packed inference is MPS-only with architecture coverage for Qwen3.5 dense and MoE (including compatible Qwen3.