Tokenization is the foundational preprocessing step that converts raw text into discrete units, called tokens, that neural networks can process. Without effective tokenization, transformer models cannot function at all, since they require fixed-size numerical representations of language.
The central challenge of tokenization lies in deciding how to split text. Character-level tokenization creates enormous sequences — for example, the word 'beautiful' becomes 8 separate tokens — while word-level tokenization fails catastrophically on out-of-vocabulary words, particularly in morphologically rich languages or when dealing with domain-specific terminology such as 'COVID-19' or 'gpt-4'.
Subword tokenization strategies, including Byte-Pair Encoding (BPE), WordPiece, and SentencePiece, address these shortcomings by learning statistical patterns from training data and splitting text into meaningful subword units. This approach balances vocabulary size — keeping it tractable for model embedding layers, typically in the range of 30,000 to 50,000 tokens — with expressiveness, allowing the tokenizer to handle rare words and neologisms gracefully.
Modern production systems rely entirely on subword tokenization. BERT uses WordPiece with a vocabulary of approximately 30,000 tokens, GPT-3 uses BPE with 50,000 tokens, and multilingual models use SentencePiece to handle more than 100 languages within a single tokenizer. The choice of tokenization strategy directly impacts model performance, inference speed, and memory consumption, making it a critical hyperparameter in production NLP pipelines.
Analogy🏏Cricket
🏏 Think of it like cricket: Imagine Virat Kohli batting in an ODI against Australia. At the start of the powerplay, fast bowlers like Pat Cummins are delivering short-pitched deliveries with aggressive fields, so Kohli pays intense attention to the pace bowlers' patterns and field placement (the bouncer-risk intel). By the 35th over, the same bowlers are tiring, spinners like Adam Zampa have come on with deeper fielders, and the match situation is different, so Kohli now focuses his attention entirely on detecting the googly and reading the turn—his attention weights shift completely to different aspects of the bowling. Just as Kohli's focus selectively weights different threats depending on the match context, the attention mechanism in transformers computes a probability distribution (attention weights) over all input tokens, assigning high weight to relevant context and low weight to irrelevant noise. The Query-Key-Value framework mirrors this perfectly: Kohli's current batting intent (Query) interacts with each bowler's recent delivery history and field setup (Key), producing a match-strength score (attention weight), and then the mechanism retrieves the most valuable tactical insight from each phase (Value). This reveals why attention works so powerfully: just as a world-class batsman dynamically reweights which aspects of the opposition matter most in each moment, neural networks using attention learn to focus computational resources exactly where the context is most predictive, making the entire system adaptive rather than fixed.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.