Mixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0

Mixtral 8x7B, a sparse mixture-of-experts language model, was published in January 2024. It outperforms or matches Llama 2 70B and GPT-3.5 on most evaluated benchmarks while using only 13B active parameters per token, and both base and instruction-tuned versions were released under the Apache 2.0 licence.

Background

Sparse mixture-of-experts, or SMoE, is an approach to scaling language models without proportionally scaling inference costs. The idea is to divide a network’s feedforward layers into separate sub-networks, called experts, and use a small router to pick only a few of them for each token. The rest sit idle. The model is large on paper but lean in practice.

What happened

Albert Q. Jiang and colleagues introduced Mixtral 8x7B, a sparse mixture-of-experts language model. Its architecture follows Mistral 7B, with one key change: each layer contains 8 feedforward blocks rather than one. For every token, at every layer, a router network selects two of those eight experts to process the current state and combine their outputs.

The routing is done per token and per layer, so the pair of experts chosen can differ from one token to the next. Each token has access to 47B parameters in total, but only 13B are active during inference. The model was trained with a context window of 32k tokens.

On benchmarks, Mixtral outperforms or matches Llama 2 70B and GPT-3.5 on most evaluated benchmarks, with the largest margins in mathematics, code generation, and multilingual work. The team also released a fine-tuned variant, Mixtral 8x7B - Instruct, trained to follow instructions. That model scores above GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and the Llama 2 70B chat model on human benchmarks available at the time.

Both models were released under the Apache 2.0 licence, imposing minimal conditions and allowing broad commercial and research use.

Why it mattered

Mixtral showed that a sparse mixture-of-experts design could give a model access to 47B parameters while only activating 13B of them at inference time, cutting compute costs without sacrificing quality. The instruction-tuned version scores above GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and the Llama 2 70B chat model on human benchmarks available at the time, and both models were released under the Apache 2.0 licence, imposing minimal conditions and allowing broad commercial and research use.

Sources

• Mixtral of Experts. arxiv.org. Primary source
• Mixtral of Experts. doi.org. Primary source

Cite this page

AI Achievements. (2024). Mixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0. Retrieved 2026-08-29, from https://achievements.ai/milestone/mixtral-8x7b-sparse-mixture-of-experts-model-released-under

@misc{achievements_mixtral_8x7b_sparse_mixture_of_experts_model_released_under,
  title  = {Mixtral 8x7B: sparse mixture-of-experts model released under Apache 2.0},
  author = {{AI Achievements}},
  year   = {2024},
  url    = {https://achievements.ai/milestone/mixtral-8x7b-sparse-mixture-of-experts-model-released-under}
}