If you run MiniMax-M3 locally, two merges that landed in llama.cpp on July 26 will affect your next llama-server start. Together, PR #24908 and PR #25113 add MiniMax Sparse Attention (MSA) and a Qwen2.5-VL style vision tower. The catch is that the vision merge renames an internal tensor, so every MiniMax-M3 GGUF produced before the change has to be regenerated before the vision path will work.
What changed in PR #24908
PR #24908, titled “Add MiniMax-M3 (MSA: MiniMax Sparse Attention) support”, was opened by contributor timkhronos and merged by maintainer ngxson on July 26. It introduces a 60-layer, 128-expert mixture-of-experts model with 3 dense layers and 57 MoE layers, and wires in MSA as the attention mechanism.
The MSA design is a GQA block-sparse scheme. A per-GQA-group indexer scores the visible causal context, max-pools scores into 128-token key blocks, and selects the top 16 blocks for each query. The query’s local block is always force-included. That works out to 16 blocks of 128 tokens, or 2048 key-value entries, per query. Because that window is fixed, decode attention cost no longer scales with context length. The PR description reports flat throughput across a wide range: “MSA decode is essentially flat with context (7.7 tok/s at 5k -> 7.67 at 62k)”.
The implementation calls ggml_flash_attn_ext directly in both batch and decode paths, so Flash Attention is required. If FA is off, the code falls back to dense attention with a warning. Unified KV cache combined with multiple sequences (--kv-unified + -np > 1) also drops back to dense, because the indexer cache assumes per-sequence KV. Multi-stream inference via --parallel N does work as long as the KV cache stays per-sequence. F16 and BF16 main KV caches are tested; quantized KV-cache types are not yet validated.
Two features are explicitly listed as unsupported: context shifting, which would desynchronize the indexer cache, and partial or middle llama_memory_seq_rm, which breaks block anchoring silently.
The PR description is emphatic about the dense fallback. “MSA is not an optional speed optimization, as the model is trained with sparse attention, so block selection is part of the model’s semantics. Running dense attention is an out-of-distribution approximation that degrades output quality.” In other words: if you turn off Flash Attention or force the unified KV cache, you will get outputs, but they are not what the model was trained to produce.
What changed in PR #25113
PR #25113, also by timkhronos and merged by ngxson on July 26, adds vision support through the mtmd (multimodal) path. The vision tower is a Qwen2.5-VL style ViT, but it does not reuse the qwen2.5vl branch as-is. Three differences matter for anyone reading the diff.
First, the rotary positional embeddings use three axes (T/H/W) instead of two, so the graph-level cos/sin are computed the same way as build_rope_2d but generalized to three axes. Second, the vision MLP is a plain GELU-erf with no gating, where the qwen2.5vl path uses a gated FFN. Third, the projector runs in two stages: a per-patch MLP, then a 2x2 group concat, then a merge MLP, both using GELU-erf, with no post-layer norm and no window attention. Only pre-layernorm is used.
The immediate user-visible change is a tensor rename. Maintainer ngxson noted in the PR thread: “small heads up: I changed a tensor name merge —> merger to make it consistent with what already existing in the code base, GGUF will need to be regenerated.” That is why existing MiniMax-M3 GGUFs break the vision path: the old files contain the old name.
What This Means
For most local users, the path of least friction is simple. If you only run text, pull the latest llama.cpp build, keep Flash Attention on, and your current MiniMax-M3 GGUF continues to work with MSA active. The performance numbers in the PR description suggest you should see roughly the same tokens per second at 5k context as at 62k, which is the main practical reason the change matters: long-context workloads stop getting quadratically more expensive.
If you run the vision path, regenerate the GGUF after updating llama.cpp. Old files will not load the vision tower under the new code because of the merge to merger rename. The two projects that build and quantize MiniMax-M3 GGUFs will need to re-run their pipelines before distributing new uploads.
Two caveats are worth keeping in mind. First, sparse attention is part of the model’s training, so any environment that disables Flash Attention or forces a unified multi-sequence KV cache will silently degrade output quality while still producing tokens. Second, the sparse path does not yet validate quantized KV caches, which is the next thing the llama.cpp maintainers will need to nail down if MiniMax-M3 is going to be useful on smaller VRAM tiers.
The Bottom Line
llama.cpp now runs MiniMax-M3 with sparse attention and vision. The work lands cleanly if you keep Flash Attention on, leave the KV cache per-sequence, and regenerate your GGUF after pulling the vision merge. For local users who care about long-context cost or on-device multimodal, this is the merge that makes MiniMax-M3 practical on consumer hardware.