Vicuna
By LMSYS
Vicuna is an open-source chatbot model released in 2023 by LMSYS (the Large Model Systems Organization), created by fine-tuning Meta's LLaMA base model on user-shared conversations collected from ShareGPT, and notable for claiming roughly…
Definition
Vicuna is an open-source chatbot model released in 2023 by LMSYS (the Large Model Systems Organization), created by fine-tuning Meta's LLaMA base model on user-shared conversations collected from ShareGPT, and notable for claiming roughly 90% of ChatGPT's response quality in early evaluations at a small fraction of the training cost.
Overview
Vicuna was developed by LMSYS, a research collective formed by researchers from UC Berkeley, Carnegie Mellon University, Stanford, and UC San Diego, shortly after Meta released LLaMA's weights to the research community in early 2023. Rather than generating synthetic instruction data (as Stanford's contemporaneous Alpaca project did), the Vicuna team fine-tuned LLaMA on roughly 70,000 real multi-turn conversations shared publicly by users on ShareGPT, a browser extension that let people share their ChatGPT conversation logs. The resulting Vicuna-13B model, released in two sizes (7B and 13B parameters), was evaluated using an early and influential methodology: having GPT-4 itself judge and score pairs of responses from different models on a shared set of questions. Using this LLM-as-judge approach, the Vicuna team reported that Vicuna-13B achieved roughly 90% of ChatGPT's quality score, a striking result given it was fine-tuned for a few hundred dollars on top of an existing open base model, and it quickly became one of the most widely used and cited open chatbot models in 2023. Vicuna's release was accompanied by FastChat, an open-source platform LMSYS built for training, serving, and evaluating chatbot models, which became widely used infrastructure in the open LLM community. LMSYS later extended this evaluation approach into the Chatbot Arena, a large-scale, crowdsourced head-to-head comparison platform where users vote on anonymized model outputs, which became one of the most referenced community leaderboards for comparing both open and closed LLMs. While newer, larger open and closed models have since surpassed Vicuna's raw capability, it remains historically significant as one of the models that demonstrated fine-tuning a modestly sized open base model on real conversational data could produce a surprisingly competent chatbot cheaply, catalyzing rapid growth in the open-source LLM ecosystem during 2023.
Key Concepts
- Fine-tuned from Meta's LLaMA base model on real ShareGPT user conversations
- Released in 7B and 13B parameter sizes by the LMSYS research collective
- Popularized GPT-4-as-judge evaluation methodology for comparing chatbot quality
- Reported roughly 90% of ChatGPT's quality at a small fraction of the training cost
- Released alongside FastChat, an open training and serving framework
- Catalyzed LMSYS's later Chatbot Arena crowdsourced model evaluation platform
- Historically significant early demonstration of cheap, effective open-model fine-tuning
- Trained on real multi-turn conversational data rather than synthetic instructions