StarCoder
By BigCode (Hugging Face & ServiceNow)
StarCoder is an open-weight large language model specialized for code generation, released in 2023 by the BigCode project — a collaboration between Hugging Face and ServiceNow — trained on permissively licensed source code from GitHub.
Definition
StarCoder is an open-weight large language model specialized for code generation, released in 2023 by the BigCode project — a collaboration between Hugging Face and ServiceNow — trained on permissively licensed source code from GitHub.
Overview
StarCoder emerged from the BigCode project's effort to build a transparent, openly licensed alternative to closed code-generation models like GitHub Copilot's underlying Codex model. It was trained on 'The Stack,' a large dataset of permissively licensed source code collected from GitHub across dozens of programming languages, with careful attention paid to filtering out code under restrictive licenses and giving developers a way to opt their code out of training — an approach meant to address licensing and provenance concerns that had been raised about earlier code models. With roughly 15 billion parameters, StarCoder supported a notably long Context Window for its time, useful for reasoning across larger files and multi-file codebases, and it came with an instruction-tuned variant better suited to following natural-language coding requests rather than only completing code. Its openness made it a popular base for fine-tuning custom coding assistants and for research into code-focused Foundation Models built on the same PyTorch tooling much of the open-source AI ecosystem relies on. BigCode followed up with StarCoder2, a next-generation version developed in collaboration with NVIDIA and trained on an expanded version of The Stack, continuing the project's focus on openness and transparency. StarCoder is often discussed alongside Meta's CodeLlama as one of the leading open-weight options specifically for code generation and completion tasks.
Key Concepts
- Open-weight code generation model released by the BigCode project in 2023
- Trained on 'The Stack,' a large permissively licensed GitHub code dataset
- Roughly 15 billion parameters with a notably long context window for its era
- Includes an instruction-tuned variant for natural-language coding requests
- Built with attention to license provenance and developer opt-out from training data
- Followed by StarCoder2, developed in collaboration with NVIDIA