# Multi-head Latent Attention

Mistral released Mistral Large 4 (Le Chonk) on Hugging Face, a multimodal Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context window, shipped as open weights in FP8 and NVFP4 variants. It pairs a large expert pool with sparse attention and a Pixtral vision encoder, so the language model selects a few experts per token while an attention indexer selects which earlier positions to attend to. On the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch it, the model scores 82%, which Mistral describes as the highest reported score on that test.

Mistral released Mistral Large 4 (Le Chonk) on Hugging Face, a multimodal Mixture-of-Experts model with about 1T total and 49B active parameters and a 1M-token context window, shipped as open weights in FP8 and NVFP4 variants. It pairs a large expert pool with sparse attention and a Pixtral vision encoder, so the language model selects a few experts per token while an attention indexer selects which earlier positions to attend to. On the Artificial Analysis Cyber Index test that asks a model to reproduce a real vulnerability in open-source software and then patch it, the model scores 82%, which Mistral describes as the highest reported score on that test.