How GPT Models Are Trained: Pretraining to RLHF

1How GPT Models Are Trained

GPT-style models are trained in a sequence of stages that transform raw internet text into a helpful assistant. The pipeline is: pretraining on a very large corpus to learn language and world knowledge, supervised fine-tuning on instruction–response pairs to teach the model to be useful, and reinforcement learning from human feedback to align its answers with human preferences.

Each stage builds on the last. Pretraining creates a broad but untamed text predictor; fine-tuning shapes it into something that answers questions; alignment makes it safer and more consistent. Understanding these steps demystifies why models behave the way they do.

2Stage 1: Pretraining

Pretraining is the compute-heavy foundation where the model learns to predict the next token across enormous volumes of text — books, code, articles, and web pages. The objective is deceptively simple: given a sequence, guess the token that comes next, and adjust weights to reduce the error.

By doing this billions of times, the model internalises grammar, facts, reasoning patterns, and even some coding ability, purely as a side effect of getting better at prediction. This stage consumes the vast majority of the total training cost and produces what is called a base model.

• Objective: next-token prediction (self-supervised — no human labels needed)
• Data scale: trillions of tokens from diverse public and licensed sources
• Output: a 'base model' that continues text but does not reliably follow instructions
• Cost: by far the most expensive stage in compute and time

3Data Curation and Tokenization

Before pretraining, raw data is cleaned, deduplicated, and filtered for quality and safety, because the model learns whatever patterns the data contains. Text is then split into tokens — sub-word chunks — by a tokenizer, so the model operates over a fixed vocabulary rather than raw characters.

4Stage 2: Supervised Fine-Tuning

Supervised fine-tuning (SFT) teaches the base model to behave like an assistant by training it on curated examples of instructions paired with high-quality responses. Human writers or reviewers produce demonstrations such as 'Summarise this article' followed by an ideal summary, and the model learns to imitate that style.

This stage is far smaller than pretraining — thousands to hundreds of thousands of examples rather than trillions of tokens — but it has an outsized effect. It is what converts a raw predictor into a model that responds to prompts, follows formatting requests, and stays on task.

• Input: instruction–response demonstrations written or curated by humans
• Effect: model learns to answer, summarise, and follow instructions
• Scale: relatively small, high-quality dataset
• Result: an 'instruct' or SFT model, much more usable than the base model

5Stage 3: RLHF

Reinforcement learning from human feedback aligns the model with what people actually prefer, going beyond imitation. Instead of showing the model a single ideal answer, RLHF shows humans several candidate answers and asks which is better, then uses those comparisons to train a separate reward model that scores responses.

• Collect comparisons: humans rank multiple model answers to the same prompt.
• Train a reward model: it learns to predict which answer humans would prefer.
• Optimise the policy: the language model is fine-tuned to maximise the reward model's score, often with PPO.
• Constrain drift: a penalty keeps the tuned model from straying too far from the SFT model.

6Newer Alignment Methods

RLHF is effective but complex, so newer methods aim to reach similar results with less machinery. Direct preference optimisation (DPO) skips the separate reward model and PPO loop, optimising the language model directly on preference pairs with a simpler objective.

Other pipelines use reinforcement learning from AI feedback, where a capable model helps generate or judge preferences, reducing the human labelling burden. These approaches are increasingly common because they cut cost and engineering complexity while keeping much of RLHF's benefit.

7Best Practices When Training or Fine-Tuning

Most practitioners will not pretrain from scratch, but many fine-tune. A few principles keep results reliable.

• Start from the strongest available base or instruct model rather than pretraining unless you truly need to.
• Prioritise data quality over quantity — a few thousand clean examples often beat a noisy million.
• Hold out an evaluation set so you can measure whether fine-tuning actually helped.
• Watch for catastrophic forgetting: aggressive fine-tuning can erode general capabilities.
• For alignment, consider DPO before full RLHF unless you have the infrastructure for a reward model and PPO.

8Key Takeaways

The training pipeline for GPT models follows a clear progression.

• Pretraining learns language and knowledge via next-token prediction on huge corpora.
• Supervised fine-tuning turns the base model into an instruction follower using curated demonstrations.
• RLHF aligns outputs with human preferences through a reward model and policy optimisation.
• Data gets smaller and cleaner at each stage while its influence per example grows.
• DPO and AI-feedback methods now offer lighter-weight alternatives to classic RLHF.

9Frequently Asked Questions

Q: What is the difference between a base model and a chat model? A: A base model only predicts the next token and tends to continue text rather than answer questions. A chat or instruct model has been fine-tuned and usually aligned, so it follows instructions and behaves like an assistant.

Q: Do I need RLHF to fine-tune a model? A: No. Many useful fine-tunes use only supervised fine-tuning on instruction data. RLHF or DPO adds alignment to human preferences, which matters most for open-ended assistant behaviour rather than narrow tasks.

Q: Why is pretraining so expensive? A: It processes trillions of tokens and updates billions of parameters many times, requiring large clusters of accelerators running for extended periods. The later stages use far less data and compute.

Q: What is DPO and how does it relate to RLHF? A: Direct preference optimisation trains the model directly on preferred-versus-rejected answer pairs without a separate reward model or reinforcement learning loop. It targets the same goal as RLHF with a simpler, often cheaper process.