Mixture Of Experts
Mixture of Experts (MoE) is a neural network architecture design in which a model is composed of many specialized sub-networks ("experts"), with a learned routing mechanism ("gate" or "router") selecting only a small subset of experts to process each input, allowing the model's total parameter count to scale up…
19 resources across 1 library
Glossary Terms(19)
DeepSeek
DeepSeek is a family of large language models developed by DeepSeek AI, a Chinese AI research company, notable for releasing highly capable models as open-weig…
Mistral
Mistral refers to the family of large language models developed by Mistral AI, a French AI company known for releasing efficient, high-performing models — incl…
Qwen
Qwen is a family of large language models developed by Alibaba Cloud, released across multiple open-weight and proprietary versions spanning general chat, codi…
Falcon
Falcon is a family of large language models developed by the Technology Innovation Institute (TII), a research institute based in Abu Dhabi, United Arab Emirat…
Llama 4
Llama 4 is Meta's next-generation open-weight model family, released in 2025, introducing a mixture-of-experts architecture and native multimodality across a r…
Mixtral 8x7B
Mixtral 8x7B is an open-weight mixture-of-experts language model released by Mistral AI in December 2023, using eight expert sub-networks per layer with only t…
DeepSeek-V3
DeepSeek-V3 is a large open-weight mixture-of-experts language model released by DeepSeek in December 2024, serving as the general-purpose foundation model tha…
Mixture of Experts (MoE)
Mixture of Experts (MoE) is a neural network architecture design in which a model is composed of many specialized sub-networks ("experts"), with a learned rout…
Sparse Mixture Model
A sparse mixture model is a neural network design in which only a small subset of the model's parameters are activated for any given input, rather than the ent…
Phi-4
Phi-4 is a small, open-weight language model from Microsoft Research, designed to deliver strong reasoning and coding performance at a fraction of the paramete…
Jamba
Jamba is a family of large language models from AI21 Labs built on a hybrid architecture combining Transformer and Mamba (state space model) layers with mixtur…
Nemotron
Nemotron is NVIDIA's family of open large language models, optimized and distilled for efficient inference on NVIDIA GPUs, spanning general-purpose, reasoning-…
Mixture of Experts
Mixture of Experts (MoE) is a neural network architecture that replaces a single dense feed-forward layer with many smaller "expert" sub-networks and a learned…
Speculative Decoding
Speculative decoding is an inference optimization technique that speeds up large language model text generation by using a smaller, faster "draft" model to pro…
Knowledge Distillation
Knowledge distillation is a machine learning technique for training a smaller "student" model to reproduce the behavior of a larger, more capable "teacher" mod…
DBRX
DBRX is an open-weight large language model released by Databricks (developed by its Mosaic AI research team) in March 2024, built as a fine-grained mixture-of…
Grok-1
Grok-1 is the first large language model developed by Elon Musk's xAI, announced in November 2023 as a 314-billion-parameter mixture-of-experts model, and late…
Gemini 1.5
Gemini 1.5 is a family of multimodal large language models from Google DeepMind, released starting February 2024 in Pro and Flash variants, notable for introdu…
Phi-3.5
Phi-3.5 is a family of small, efficient language and multimodal models from Microsoft, including a 3.8B-parameter mini text model, a mixture-of-experts variant…