Meta opens an 816B coding-agent model: compressed MoE and hybrid attention cut long-session KV cache about 3.7x
Synopsis
Meta released the weights of Muse Spark Open, an 816B-parameter omnimodal mixture-of-experts model with 42.7B active at inference, co-trained with Muse Code for coding-agent workflows. A compressed MoE block and a hybrid global/local attention schedule cut inter-GPU traffic and long-session KV cache memory, and the model supports a 1M-token context with text, image, video, and audio input. On Meta-reported benchmarks it is competitive with frontier-lab models on agentic coding and knowledge work, though each model was tested with its own lab's coding agent rather than a shared harness.
Interpretation
Coding agents no longer have to work from text alone; screenshots, screen recordings, and speech can enter the fix-and-test loop directly. Earlier coding agents were mainly text-facing, so image, video, or audio had to be converted to text first. Meta describes three uses: comparing a rendered-page screenshot against a design mockup or bug-report image and fixing layout code, watching a bug screen capture and writing a reproduction script or end-to-end test, and voice-driven pair programming where audio is taken as direct input rather than transcribed first.
The model compresses the input vector before handing it to the experts, shrinking each expert while the router still reads the full-width vector. An uncompressed mixture-of-experts design lets expert parameters and inter-GPU traffic grow with total parameter count. A shared pooler compresses the input vector before the experts, and a shared unpooler projects their combined output back to the original dimension; by Meta's account this halves each expert's parameters, shrinks total model size by roughly 800B parameters versus an uncompressed design, and reduces inter-GPU traffic in the MoE layer.
Long-context memory cost is split between local and global attention, with local layers keeping only a recent window and older information held in the global layers. A conventional model keeps its own KV cache in every layer, so cost grows with context length. Of 62 layers, 46 are local attention with a 2048-token sliding window and rotary position embeddings, and 16 are global attention with no positional encoding; at 128K-token context, KV cache memory is about 3.7x lower than in models with only global attention layers.
Serving uses 4-bit weights to free memory for more concurrent sessions, letting the full model fit on a single node. Weight memory footprint caps how many agent sessions fit on a node. The NVFP4 checkpoint stores expert weights in 4-bit and keeps full bf16 only for attention weights and the experts in the first and last layers, reducing weight size from roughly 1.6 TB to 540 GB so the full model fits on a single 4×GB300 node without multi-node interconnect.
Perspective
This report is for researchers and engineers who deploy coding agents, and covers the open-weight release, the architecture, the memory and throughput figures, and the serving path. By Meta's account, Muse Spark Open should feel close to flagship closed models in routine agent loops, is expected to need more guidance or retries than Claude Opus 5 or GPT-5.6 Terra on large multi-file changes across many steps, holds up well against Grok 4.5 and Gemini 3.6 Flash, and beats every model in the chart except Opus 5 when an agent has to write up its work.
The benchmark scores are reported by Meta, and each current model was tested with its own lab's coding agent, Muse Spark with Muse Code, Claude Opus 5 with Claude Code, and GPT-5.6 Terra with Codex, rather than a shared harness, so the scores reflect each model in the tooling it was built for. The roughly 3.7x KV cache reduction is measured at 128K-token context against models with only global attention layers, and the drop from roughly 1.6 TB to 540 GB corresponds to the NVFP4 checkpoint.