Meta ships an 816B open-weight multimodal model: 256 experts, 8 active per token, 1M-token context
Synopsis
Meta released Muse Spark Open, an open-weight model that puts text, images, audio and video into one mixture-of-experts model of roughly 816 billion total parameters and about 42 billion active per token, with a context window of up to one million tokens and text as output. It is trained for agentic coding, so it can read a codebase, write and debug code and call tools, while also taking a screenshot, a voice note or a screen recording as input. Compared with Muse Glimmer, the small dense model from the same series released in August, this scales the mixture-of-experts design up and ships BF16 and NVFP4 checkpoints with llama.cpp, vLLM and SGLang recipes.
Interpretation
An open-weight model now brings text, images, audio and video into a single model, so developers get one interface for asking questions about different kinds of input. Muse Glimmer, released in August, is a small dense model from the same series meant for local running, and the two share the same tokenizer and vocabulary. The model has roughly 816 billion total parameters, about 42 billion active per token, a context window of up to one million tokens, and text output.
The model is trained for agentic coding: reading a codebase, writing and debugging code, and calling tools, planning, acting, observing and self-correcting over long tasks. Training draws on real developer workflows such as multi-file refactors and long debugging sessions. It also accepts images, audio and video while working on code, so the same model can read a screenshot, listen to a voice note or watch a screen recording.
Each layer holds 256 experts and a router selects 8 per token, with experts holding about 97% of all parameters, so every expert stays in memory while each token is computed with about 42 billion parameters. Muse Glimmer uses one dense feed-forward network per layer, and Muse Spark Open replaces it with a mixture-of-experts block. The language model has 62 layers and a hidden dimension of 8192, with 128 query heads and 16 key-value heads per layer; the BF16 checkpoint is about 1.5 TB and the NVFP4 checkpoint stores parameters in 4-bit precision for NVIDIA Blackwell GPUs.
A perception encoder turns images and video frames into tokens, and an audio encoder emits one audio token every 80 milliseconds, or 12.5 tokens per second. Video input goes through the same encoder as images, while audio passes through an audio tokenizer and is marked by audio tokens in the chat template. The perception encoder has 1.8 billion parameters, 50 layers, a hidden dimension of 1536 and 16 attention heads, splitting each image into 14x14 patches and producing 4096-dimensional features.
Perspective
The report covers the model release and its deployment and fine-tuning notes: roughly 816 billion total parameters, about 42 billion active per token, a context window of up to one million tokens, BF16 and NVFP4 checkpoints, and reference configurations for llama.cpp, vLLM, SGLang, MLX and Inference Endpoints. The fine-tuning example trains for 500 steps on 8 nodes of 8 H100 GPUs, about 2 hours at about 8,000 tokens per second, with mean token accuracy rising from 72% to 84% and loss falling from 1.26 to 0.57, for that example task and dataset only.
The model is described as the most capable US open-weights model to date, a claim from the release itself. Reasoning cannot be turned off, the default level is xhigh, and only low reduces token use. Video is sampled at 2 FPS by default, capped at 96 frames, and raising the frame rate increases memory use and prefill latency. The NVFP4 accuracy loss is described as negligible, but that checkpoint targets NVIDIA Blackwell GPUs.