Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
Definition
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
Overview
Before a language model can process text, the raw string must be converted into a numeric format the model can operate on. Tokenization is the first step in this pipeline: it splits input text into discrete units, and each unique token is mapped to an integer ID from the model's vocabulary. Most modern large language models use subword tokenization algorithms, such as byte-pair encoding (BPE), which split text into pieces smaller than whole words for rare terms while keeping common words as single tokens — this balances vocabulary size against the model's ability to represent any possible input, including unusual words, code, and multiple languages. Once tokenized, each token ID is converted into a numeric embedding vector, which is then processed by the model's neural network layers, including its attention mechanism layers. The number of tokens a model can process at once is bounded by its context window, and providers typically price API usage per token, making tokenization directly relevant to both technical performance and cost. Tokenization choices affect a model's efficiency and behavior in subtle ways — languages with less common vocabulary, code with unusual syntax, and numbers can all be tokenized less efficiently than common English text, which is one reason token counts can vary meaningfully across languages for equivalent content.
Key Concepts
- Converts raw text into discrete units (tokens) a model can process
- Most modern LLMs use subword tokenization like byte-pair encoding
- Each token maps to a numeric ID from the model's fixed vocabulary
- Token counts determine context window usage and API pricing
- Balances vocabulary size against ability to represent any input text
- Efficiency varies across languages, code, and numeric content
Use Cases
Frequently Asked Questions
From the Blog
Tokenization Explained: How LLMs Read Text
LLMs do not read words or letters; they read tokens, the chunks text is split into. Learn what tokens are and why they shape cost, limits, and behavior.
Read More AI & TechnologyHow AI Tokenization Works Explained Simply
Tokenization splits text into tokens, the small chunks a language model actually reads. It drives context limits, cost, and even how models spell and count.
Read More AI & TechnologyTokenization in NLP: How Machines Break Down Text
Tokenization is the process of splitting text into smaller units a model can process, and it is the first step in nearly every NLP pipeline. This guide explains how tokenization works, common strategies, and why it matters for language models.
Read More AI & TechnologyHow tokenization affects model behaviour, cost and context limits
Tokenization explains arithmetic slips, mangled rare words and uneven costs across languages. Learn to inspect the token stream and design prompts around it.
Read More