How Large Language Models Actually Work

1What Is a Large Language Model?

A large language model is a neural network trained to predict the next token in a sequence. That's it. The "large" refers to scale: billions of parameters trained on trillions of tokens of text.

The emergent result of this scale is a system that appears to understand language, reason, write, and answer questions — because predicting text at scale requires implicit models of grammar, facts, logic, and world knowledge.

Understanding what LLMs are (and aren't) makes you a better user, a better prompt engineer, and a more thoughtful AI application developer.

2Step 1: Tokenisation

LLMs don't process characters or words — they process tokens. A tokeniser splits text into sub-word pieces according to a learned vocabulary (typically 32k–200k tokens).

The tokenisation choices have real consequences: rare words, languages with complex morphology (like Tamil or Finnish), and technical jargon may require more tokens per concept, making them relatively more expensive and less efficiently modelled than common English words.

# Using the tiktoken library (OpenAI's tokeniser)
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
text = "SkillVeris teaches coding"
tokens = enc.encode(text)
print(tokens) # [50560, 548, 285, 22816, 49625]
print(len(tokens)) # 5 tokens for 4 words
# Common words are single tokens; rare words split into sub-pieces
# "unhappy" might become ["un", "happy"]
# "SkillVeris" might become ["Skill", "Veris"]

3Step 2: Embeddings

Each token ID maps to a vector — a list of numbers (typically 768–4096 dimensions in large models) called an embedding. These embeddings are learned during training so that semantically similar tokens end up geometrically close in the embedding space.

The embedding layer is essentially a lookup table: token ID 42 maps to a specific 768-dimensional vector. Initially random; after training, the vector for "king" minus the vector for "man" plus the vector for "woman" points toward the vector for "queen" — the geometry of the embedding space captures relationships.

# Conceptually (simplified)
embedding_table = {
42: [0.21, -0.45, 0.89, ...], # 768 numbers for token 42
7: [-0.11, 0.73, 0.14, ...], # 768 numbers for token 7
}
# A sequence of 5 tokens becomes a 5x768 matrix

4The Transformer Architecture

The transformer, introduced in the 2017 paper "Attention Is All You Need," is the architecture underlying every modern LLM. A transformer consists of stacked layers, each containing several core components.

A large model like Claude or GPT-4 has 80-100+ such layers. Each pass through a layer adds nuance and context to the token representations.

• Multi-head self-attention — lets each token "look at" all other tokens in the context and decide how much to borrow from each.
• Feed-forward network (FFN) — applies a two-layer neural network to each position independently, adding non-linearity.
• Layer normalisation — stabilises training by normalising activations.
• Residual connections — add the layer's output to its input, allowing gradients to flow through deep networks without vanishing.

5Self-Attention: The Core Mechanism

Self-attention is what makes transformers revolutionary. For each token, it computes how much attention to pay to every other token in the context.

Intuitively: when processing the word "it" in "The dog chased the cat because it was fast," the attention mechanism learns to assign high attention weight to "cat" (the correct referent), solving the coreference problem implicitly.

Multi-head attention runs this process in parallel with different learned Q/K/V projections, allowing the model to attend to different types of relationships simultaneously (syntactic, semantic, positional).

• Each token creates three vectors: Query (Q), Key (K), and Value (V).
• For token A, the attention score toward token B = dot product of A's Query with B's Key.
• Scores are normalised (softmax) to sum to 1 — these are attention weights.
• Token A's output = weighted sum of all Value vectors, where weights are the attention scores.

6Feed-Forward Layers and Residuals

After attention, each token's representation passes through a position-wise feed-forward network — two linear transformations with a non-linear activation (GELU or ReLU) in between. This is where much of the model's "knowledge" is stored: the FFN weights act as a key-value memory that maps input patterns to stored facts.

Research has shown that factual recall ("Paris is the capital of France") is largely mediated by specific FFN neurons, while the attention mechanism handles contextual reasoning. Both are essential.

Residual connections add the original input to the output of each sub-layer: output = sublayer(x) + x. This ensures that information from early layers is preserved throughout the deep network and gradients can flow back without vanishing.

7How Models Are Trained

Pre-training uses self-supervised learning: no human labels are needed. The model sees a sequence of tokens, tries to predict the next one, compares its prediction to the actual next token, and updates its weights to reduce the error.

Repeated across trillions of token positions from the entire web, books, and code, this builds a rich statistical model of language and world knowledge.

Training a frontier LLM from scratch costs tens of millions of dollars in compute. This is why most AI engineers use pre-trained models via API rather than training from scratch.

• Pre-training · Trillions of tokens, web-scraped · Predict next token
• Supervised fine-tuning · Thousands of high-quality Q&A pairs · Follow instructions
• RLHF · Human preference rankings · Align with human values

8RLHF: Making Models Helpful

Reinforcement Learning from Human Feedback (RLHF) is how base pre-trained models become helpful chat assistants.

RLHF is what transforms a "predict the next token" model into a model that tries to be helpful, harmless, and honest. Constitutional AI (Anthropic's approach with Claude) extends this by using the model itself to generate and rate responses against a set of principles, reducing the need for human annotation at scale.

• Generate multiple responses to the same prompt.
• Human raters rank the responses from best to worst.
• Train a reward model to predict which response a human would prefer.
• Use reinforcement learning to tune the language model to maximise the reward model's score.

9Generating Text: Sampling Strategies

When an LLM generates text, at each step it produces a probability distribution over its entire vocabulary (~50k-200k tokens). The sampling strategy determines which token is selected.

Most production APIs combine temperature and top-p. The defaults (temperature ~1.0, top-p ~0.95) produce fluent, varied, non-repetitive text for most tasks.

• Greedy — always pick the most probable token. Deterministic but often repetitive and bland.
• Temperature — divide logits by temperature before softmax. Temperature < 1 makes the distribution sharper (more predictable); > 1 makes it flatter (more random). Temperature = 0 is greedy.
• Top-p (nucleus) sampling — sample only from the top-p fraction of the probability mass. E.g. top-p=0.9 samples from the smallest set of tokens accounting for 90% of probability.
• Top-k — sample from the k most probable tokens only.

10Context Windows

The context window is the maximum number of tokens the model can process in a single forward pass. It determines how much text the model can "see" at once.

In 2026, most frontier models have 128k-200k token context windows, with some (Gemini 1.5) supporting 1M+. Larger windows enable long document Q&A without chunking, but longer contexts are slower and more expensive to process.

• 4,096 tokens · ~3,000 words (~5-6 pages)
• 32,768 tokens · ~25,000 words (a short novel)
• 128,000 tokens · ~100,000 words (a full novel)
• 1,000,000 tokens · ~750,000 words (multiple books)

11Why Scale Matters

The "scaling laws" discovered by Kaplan et al. (2020) and Hoffmann et al. (2022) showed that model performance improves predictably as you scale model size (parameters), dataset size (tokens), and compute. Beyond a certain scale, new capabilities "emerge" — the model can suddenly do arithmetic, write code, or reason in ways smaller models cannot, even without being explicitly trained for those tasks.

This is why the race to larger models matters: more parameters + more data + more compute = qualitatively better capabilities, not just marginally better. The emergent capabilities are what drive commercial value and, increasingly, safety concerns.

12Key Takeaways

The essentials to remember about how large language models work and behave.

• An LLM is a next-token predictor: it models P(next token | all previous tokens).
• The transformer's attention mechanism lets every token attend to every other token in context, enabling long-range reasoning.
• Pre-training on massive text gives base knowledge; fine-tuning and RLHF make the model helpful and aligned.
• Context window = how much the model can "see" at once; temperature = how creative vs predictable the output is.
• LLMs don't "know" facts in the human sense — they model the statistical structure of language, which implicitly encodes facts, reasoning patterns, and world knowledge.

13What to Learn Next

Go deeper into LLMs with these follow-on topics.

• Prompt Engineering Guide — knowing how LLMs work makes you a better prompt writer.
• RAG Explained — how to ground LLM outputs in your own data.
• Fine-Tuning LLMs — adapt a pre-trained model to your specific use case.

14Frequently Asked Questions

Do LLMs actually "understand" language? This is contested. LLMs process language statistically and can perform tasks that look like understanding (answering questions correctly, translating, summarising). Whether this constitutes genuine understanding in a philosophical sense is debated. Practically: LLMs are extremely useful but make errors that suggest they don't have the grounded world understanding humans do. Calibrated scepticism is appropriate.

Why do LLMs hallucinate? Hallucination happens because LLMs are trained to produce fluent, plausible text — not to retrieve verified facts. When a model doesn't "know" something, it doesn't have a reliable "I don't know" signal; instead it generates text that continues to sound plausible, which may be wrong. RAG (grounding in retrieved documents) is currently the most effective mitigation.

What is the difference between GPT-4 and Claude? Both are large transformer-based language models trained on similar datasets with RLHF-style alignment. They differ in training objectives (Anthropic uses Constitutional AI; OpenAI uses RLHF), model architecture details, context window sizes, and system prompt behaviour. Both are highly capable; differences in practice are often task-specific. Benchmark both on your actual use case rather than relying on general rankings.

Can LLMs reason? LLMs exhibit behaviour that looks like reasoning: solving multi-step maths, writing code, explaining their thinking. However, they're also prone to systematic errors on tasks that require robust logical reasoning (especially when the surface form doesn't match training distribution). Chain-of-thought prompting improves reasoning significantly. The field of "LLM reasoning" is active research — the honest answer is "sometimes, unreliably, and we don't fully understand when."