Mixtral 8x7B
11 December 2023
Eight experts, two active per token. GPT-3.5 quality with open weights.
Abstract
Today, the team is proud to release Mixtral 8x7B, a high-quality sparse mixture of experts model (SMoE) with open weights. Licensed under Apache 2.0. Mixtral outperforms Llama 2 70B on most benchmarks with 6x faster inference. It is the strongest open-weight model with a permissive license and the best model overall regarding cost/performance trade-offs. In particular, it matches or outperforms GPT3.5 on most standard benchmarks.
Sources: AI4U editorial, Epoch AI, Hugging Face.