Tokenisation and word embeddings — Deep Learning & Neural Networks

Neural networks operate on numbers, not text. Tokenisation converts raw text into sequences of integers, and word embeddings convert those integers into dense floating-point vectors that capture semantic relationships. The choice of tokenisation strategy has a profound impact on model quality: character-level tokenisation (one integer per character) produces very long sequences that are expensive for transformers; word-level tokenisation creates vocabulary gaps for rare or unknown words; subword tokenisation (Byte Pair Encoding, WordPiece, SentencePiece) balances sequence length and vocabulary coverage by breaking rare words into frequent subword units. Modern language models use subword tokenisation with vocabularies of 32,000–100,000 tokens. Word embeddings, once trained, encode rich semantic structure: synonyms appear close in embedding space, analogies like 'king - man + woman ≈ queen' are captured as vector arithmetic, and multilingual embeddings align words with similar meanings across languages. Understanding tokenisation and embeddings is essential because they determine the model's input format, vocabulary coverage, and the quality of the semantic representations that downstream tasks rely on.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.