Fine-tuning BERT for text classification — Deep Learning & Neural Networks

BERT (Bidirectional Encoder Representations from Transformers), released by Google in 2018, fundamentally changed NLP by demonstrating that a single pre-trained model fine-tuned on task-specific data could surpass specialised architectures trained from scratch on every major NLP benchmark simultaneously. BERT is pre-trained on two objectives: Masked Language Modelling (MLM — predict randomly masked tokens from both left and right context) and Next Sentence Prediction (NSP — predict if two sentences are consecutive). This bidirectional pre-training means every token's representation is informed by both its preceding and following context — unlike GPT which is purely left-to-right. For text classification, BERT appends a special [CLS] token at the start of every input; the final hidden state of the [CLS] token captures an aggregate sequence representation that is fed to a classification head. Fine-tuning updates all BERT parameters simultaneously with the classification head using a small learning rate (2e-5 to 5e-5) for 3–5 epochs — dramatically fewer than training from scratch. Understanding BERT fine-tuning is the template for fine-tuning any encoder transformer (RoBERTa, DistilBERT, ALBERT, DeBERTa) on any text classification task.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.