Orca (model)
Orca is a series of research language models from Microsoft that use explanation-tuning — training on step-by-step reasoning traces distilled from a larger teacher model like GPT-4 — to teach smaller models more robust reasoning ability…
Definition
Orca is a series of research language models from Microsoft that use explanation-tuning — training on step-by-step reasoning traces distilled from a larger teacher model like GPT-4 — to teach smaller models more robust reasoning ability than standard instruction tuning produces.
Overview
Orca was introduced by Microsoft Research in 2023 as an investigation into why smaller open models fine-tuned via imitation learning (training on a larger model's final outputs) tend to mimic surface style without acquiring the underlying reasoning capability. The Orca paper argued that prior small-model instruction-tuning efforts, which trained on short input-output pairs from a teacher model, taught models to sound like GPT-4 without teaching them to reason like it. Orca's fix was "explanation tuning": instead of training only on a teacher model's final answer, the training data includes detailed, step-by-step explanations of how the teacher arrived at that answer, elicited through system prompts that ask the teacher (GPT-4, and initially ChatGPT/GPT-3.5) to "think step by step," justify its reasoning, and break down complex tasks. A 13-billion-parameter Orca model trained this way on roughly five million explanation traces showed substantial improvements on reasoning benchmarks (BigBench Hard, complex zero-shot tasks) compared to prior instruction-tuned open models of similar size, and narrowed the gap to much larger proprietary models on some tasks. Orca 2, released in late 2023, extended this idea with "Cautious Reasoning": rather than always eliciting the same style of step-by-step chain of thought, the model was trained to select from multiple reasoning strategies (direct answer, step-by-step, recall-then-generate) depending on the task, since not every problem benefits from lengthy explicit reasoning. Orca 2 was released in 7B and 13B sizes, fine-tuned from LLaMA 2, and Microsoft published the weights for research use, though with a research-only license rather than fully permissive open-source terms. Orca's broader significance is methodological: it demonstrated that distillation quality — specifically, distilling reasoning process rather than just final answers — matters more than distillation scale for producing small models with strong reasoning, an idea that subsequently influenced other explanation-tuning and reasoning-distillation efforts across the open-model ecosystem.
Key Concepts
- Explanation-tuning: trains on step-by-step reasoning traces, not just final answers
- Distilled from larger teacher models (GPT-4 and GPT-3.5/ChatGPT)
- Orca 2 introduces Cautious Reasoning, choosing among multiple reasoning strategies per task
- Fine-tuned on top of LLaMA-family base models
- Released in 7B and 13B parameter sizes
- Strong performance on reasoning benchmarks relative to similarly sized open models
- Research-license weights published by Microsoft Research
- Influenced the broader field's approach to distillation-based reasoning transfer