Practice — sentiment analysis with BERT — Deep Learning & Neural Networks

What You'll Build

You will build a complete cricket commentary sentiment analysis system using BERT fine-tuning via HuggingFace Transformers. The system classifies IPL match commentary into three sentiment categories: positive (batting milestones, brilliant shots, match-winning performances), negative (dismissals, collapses, costly errors), and neutral (routine deliveries, match administration, weather events). The project covers the complete production NLP pipeline: dataset creation and tokenisation, two-phase training (frozen base then full fine-tuning), evaluation with per-class metrics, zero-shot baseline comparison, and model export for deployment. This exercise consolidates every M5 concept: transformer architecture, tokenisation, HuggingFace API, BERT fine-tuning, and zero-shot learning — all applied to a unified cricket NLP task. The completed model can be deployed as a real-time commentary tagger for IPL broadcasts, automating sentiment labelling at the speed of the match.

Analogy🏏Cricket

🏏 Think of it like cricket: Building a neural net from scratch is like a young cricketer learning the game by playing gully cricket with a tape-ball before ever stepping into a coaching academy. You don't use a bowling machine, you don't get video analysis, you don't have a structured training manual — you just play, make mistakes, adapt. Just as gully cricket teaches the core instincts of timing, footwork, and reading the ball that no academy drill can fully replicate, building a network from NumPy teaches the core mechanics of gradient flow, matrix shapes, and numerical stability that no framework hides. Just as every Indian international cricketer traces their instincts back to gully cricket roots, every deep learning practitioner benefits from having once implemented backpropagation themselves. The insight is that frameworks automate what you understand — building from scratch ensures you actually understand it.

Prerequisites

HuggingFace Transformers: AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
BERT fine-tuning recipe: lr=2e-5, 3–5 epochs, warmup_ratio=0.1, weight_decay=0.01
datasets library: Dataset.from_pandas(), map() for tokenisation, set_format('torch')
Zero-shot classification pipeline: pipeline('zero-shot-classification', model='facebook/bart-large-mnli')
Evaluation: classification_report, confusion_matrix from sklearn.metrics

Setup & Project Structure

This project requires the transformers, datasets, torch, scikit-learn, and pandas libraries. The pipeline has five steps: dataset generation and preparation, tokenisation, model building and training, zero-shot vs fine-tuned comparison, and final evaluation with model export. All steps are self-contained and can be run in Colab with GPU for fastest training. The complete project takes approximately 15–20 minutes on GPU.

Analogy🏏Cricket

🏏 Think of it like cricket: Before a net session you lay out the ground in clear zones — the bowling machine and pitch (data generation of synthetic IPL stats), the batsman's technique broken into stance, backlift, and follow-through (the NeuralNet forward, backward, and update methods), the throwdown routine that repeats and tracks improvement (the training loop with loss and accuracy monitoring), and finally a match simulation to test readiness (test-set evaluation). Just as a coach keeps the whole drill on one ground so a player can see how each phase connects, keeping all the code in one file lets you trace how a forward pass flows into a gradient and then a weight update. Just as you need only bat, ball, and pitch — not a full stadium — for a productive net, this exercise needs only NumPy, no heavy libraries. The payoff: a clean, well-sectioned practice structure means every mechanic is visible and debuggable, so you understand exactly why the network learns rather than treating it as a black box.

bash

# pip install transformers datasets torch scikit-learn pandas
from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    TrainingArguments, Trainer, EarlyStoppingCallback,
    pipeline as hf_pipeline
)
from datasets import Dataset
import pandas as pd, numpy as np, torch
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

np.random.seed(42); torch.manual_seed(42)
print(f'GPU available: {torch.cuda.is_available()}')

# Label mapping
LABEL2ID = {'positive': 0, 'negative': 1, 'neutral': 2}
ID2LABEL  = {0: 'positive', 1: 'negative', 2: 'neutral'}
NUM_LABELS = 3
MODEL_NAME = 'distilbert-base-uncased'  # 66M params, fast

Step 1 — Foundation

Step 1 creates the cricket commentary sentiment dataset and tokenises it for BERT. The dataset has three balanced classes: 150 positive commentaries (batting milestones, match-winning shots), 150 negative commentaries (wickets, collapses, errors), and 150 neutral commentaries (routine deliveries, toss, weather). Balance is critical — BERT fine-tuned on imbalanced data learns to predict the majority class for ambiguous examples. The tokeniser converts each commentary string into input_ids (token sequences) and attention_mask (1 for real tokens, 0 for padding) with a fixed max_length=128 truncation.

Analogy🏏Cricket

🏏 Think of it like cricket: Before the first ball is bowled, the groundsman prepares the pitch — the pitch condition is the foundation that determines what is possible. Bad pitch preparation (wrong initialisation) makes even the best bowlers and batsmen ineffective. Just as a pitch that is too green (over-seamed) or too dry (over-spun) constrains the entire match, bad weight initialisation constrains the entire training run. He initialisation is the 'neutral pitch' — well-prepared, giving both batting and bowling a fair contest, from which any outcome is possible.

python

import pandas as pd, numpy as np
from transformers import AutoTokenizer
from datasets import Dataset

np.random.seed(42)

# ── Generate cricket sentiment dataset ───────────────────────
positive = [
    'Rohit Sharma brings up his century with a magnificent six over long-on',
    'Virat Kohli plays a textbook cover drive to reach fifty in style',
    'MS Dhoni finishes the game with a helicopter shot — India wins!',
    'Suryakumar Yadav plays a stunning reverse sweep for six',
    'Jasprit Bumrah takes a hat-trick to seal a famous victory',
    'India reaches the target with 10 balls to spare — clinical performance',
    'KL Rahul plays a gorgeous flick off his legs for four',
    'Hardik Pandya smashes three consecutive sixes to win the match',
] * 19  # 152 positive samples

negative = [
    'Virat Kohli is dismissed for a duck — massive blow to India',
    'Jasprit Bumrah concedes 20 runs in the 19th over — costly collapse',
    'India lose three wickets in one over — stunning collapse',
    'Rohit Sharma is caught at the boundary — costly shot selection',
    'The team loses the match by 50 runs — complete batting failure',
    'Another run out — terrible communication between the batsmen',
    'Dropped catch proves costly as the batsman goes on to score fifty',
    'India bowled out for 89 — worst batting performance of the season',
] * 19  # 152 negative samples

neutral = [
    'The toss has been won by India who choose to bat first',
    'The match has been stopped due to a wet outfield',
    'The umpire signals a dead ball after the delivery hits the batsman',
    'The fielding restrictions are now lifted as the powerplay ends',
    'A dot ball — good length delivery outside off stump, left alone',
    'The drinks break is taken at the end of the 10th over',
    'The match referee announces a 5-minute delay due to ground inspection',
    'Maiden over completed — 6 dot balls from Bumrah',
] * 19  # 152 neutral samples

texts  = positive[:150] + negative[:150] + neutral[:150]
labels = [0]*150 + [1]*150 + [2]*150

df = pd.DataFrame({'text': texts, 'label': labels})
df = df.sample(frac=1, random_state=42).reset_index(drop=True)
print(f'Dataset: {len(df)} samples, {df["label"].value_counts().to_dict()}')

# ── Tokenise ───────────────────────────────────────────────────
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def tokenize_fn(examples):
    return tokenizer(
        examples['text'],
        truncation=True,
        max_length=128,
        padding='max_length'
    )

# Stratified split: 70/15/15
train_df, temp_df = train_test_split(df, test_size=0.30, stratify=df['label'], random_state=42)
val_df,   test_df = train_test_split(temp_df, test_size=0.50, stratify=temp_df['label'], random_state=42)

for name, split in [('Train',train_df),('Val',val_df),('Test',test_df)]:
    print(f'{name}: {len(split)} samples')

def to_hf_dataset(dataframe):
    ds = Dataset.from_pandas(dataframe.rename(columns={'label':'labels'}).reset_index(drop=True))
    ds = ds.map(tokenize_fn, batched=True).remove_columns(['text'])
    ds.set_format('torch')
    return ds

train_ds = to_hf_dataset(train_df)
val_ds   = to_hf_dataset(val_df)
test_ds  = to_hf_dataset(test_df)
print(f'\n[CLS]=101 at pos 0: {train_ds[0]["input_ids"][0].item()==101}')
print(f'Attention mask sum (= real tokens): {train_ds[0]["attention_mask"].sum().item()}')

Step 2 — Core Logic

Step 2 builds the model and establishes the zero-shot baseline before fine-tuning. The zero-shot baseline using facebook/bart-large-mnli gives us the accuracy achievable with no cricket-specific training data — this is the benchmark the fine-tuned model must beat to justify the labelling effort. The fine-tuned model uses DistilBERT with a 3-class classification head, trained with the standard BERT recipe augmented with EarlyStoppingCallback.

Analogy🏏Cricket

🏏 Think of it like cricket: The NeuralNet class is the team management system — it tracks every player's contribution (forward pass caching), attributes match outcomes to specific decisions (backward pass gradients), and adjusts each player's role for the next match (weight update). Just as the team manager must record which batsman faced which bowler and how many runs were scored (cache activations) before attributing success or failure, the neural net must cache z and a at every layer before computing gradients. The momentum in SGD is the team's institutional memory — great past performances inform future selection policy, not overriding current evidence but weighted alongside it.

python

from transformers import (
    AutoModelForSequenceClassification, TrainingArguments, Trainer,
    EarlyStoppingCallback, pipeline as hf_pipeline
)
import numpy as np
from sklearn.metrics import classification_report

# ── Zero-shot baseline ─────────────────────────────────────────
print('=== Zero-Shot Baseline ===')
zs_pipe = hf_pipeline('zero-shot-classification',
                       model='facebook/bart-large-mnli', device=-1)

candidate_labels = [
    'positive cricket event: batting milestone or match winning moment',
    'negative cricket event: wicket dismissal or batting collapse',
    'neutral cricket administration or routine delivery'
]

# Evaluate on test set (first 30 for speed)
test_texts  = [test_df.iloc[i]['text'] for i in range(30)]
test_labels = [test_df.iloc[i]['label'] for i in range(30)]

zs_preds = []
for text in test_texts:
    r     = zs_pipe(text, candidate_labels=candidate_labels)
    top   = r['labels'][0]
    pred  = 0 if 'positive' in top else (1 if 'negative' in top else 2)
    zs_preds.append(pred)

zs_acc = np.mean(np.array(zs_preds) == np.array(test_labels))
print(f'Zero-shot accuracy: {zs_acc:.1%}')
print(classification_report(test_labels, zs_preds, target_names=['positive','negative','neutral']))

# ── Fine-tuned model ───────────────────────────────────────────
print('\n=== Fine-Tuning DistilBERT ===')
model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    num_labels=NUM_LABELS,
    id2label=ID2LABEL,
    label2id=LABEL2ID
)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {'accuracy': (preds == labels).mean()}

args = TrainingArguments(
    output_dir='/tmp/cricket_sentiment',
    num_train_epochs=5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    weight_decay=0.01,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    metric_for_best_model='accuracy',
    fp16=torch.cuda.is_available(),
    logging_steps=20,
    report_to='none'
)
trainer = Trainer(
    model=model, args=args,
    train_dataset=train_ds, eval_dataset=val_ds,
    compute_metrics=compute_metrics,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=2)]
)
trainer.train()

Step 3 — Integration & Enhancement

Step 3 evaluates the fine-tuned model on the test set and compares it against the zero-shot baseline, then plots the training curves and confusion matrix. The comparison table quantifies the value of fine-tuning. The confusion matrix reveals systematic misclassifications — for cricket sentiment, neutral vs negative is typically the hardest boundary (a maiden over is neutral; a collapse starts as individual negatives but the neutral-to-negative transition depends on context).

Analogy🏏Cricket

🏏 Think of it like cricket: Mini-batch training is like a cricket coaching session where the coach evaluates the batsman against a random set of 32 deliveries from the session's total pool, not all 500 deliveries at once. Evaluating all 500 deliveries before giving any feedback (full-batch) is slow and ignores the fact that early corrections from the first 32 balls can already improve performance on balls 33–64. Mini-batch feedback (stochastic gradients) allows the batsman to improve continuously throughout the session, converging to good technique faster than waiting for the end-of-session review. The randomness (shuffle each epoch) ensures the coach doesn't accidentally train the batsman to handle only one type of delivery sequence.

python

import numpy as np
from sklearn.metrics import classification_report, confusion_matrix
import matplotlib.pyplot as plt

# ── Test set evaluation ────────────────────────────────────────
results = trainer.evaluate(eval_dataset=test_ds)
print(f'Fine-tuned test accuracy: {results["eval_accuracy"]:.2%}')

# Detailed per-class metrics
all_preds, all_labels = [], []
for batch in torch.utils.data.DataLoader(test_ds, batch_size=32):
    with torch.no_grad():
        out = trainer.model(
            input_ids=batch['input_ids'],
            attention_mask=batch['attention_mask']
        )
    preds = out.logits.argmax(dim=-1).cpu().numpy()
    all_preds.extend(preds)
    all_labels.extend(batch['labels'].cpu().numpy())

print('\nPer-class Report:')
print(classification_report(all_labels, all_preds,
                             target_names=['positive','negative','neutral']))

# ── Summary comparison ─────────────────────────────────────────
ft_acc = np.mean(np.array(all_preds) == np.array(all_labels))
print(f'\n=== Model Comparison ===')
print(f'Zero-shot accuracy:    {zs_acc:.1%}')
print(f'Fine-tuned accuracy:   {ft_acc:.1%}')
print(f'Improvement:           +{(ft_acc-zs_acc)*100:.1f}pp')

# ── Confusion matrix ───────────────────────────────────────────
cm = confusion_matrix(all_labels, all_preds)
fig, ax = plt.subplots(figsize=(7,6))
im = ax.imshow(cm, cmap='Blues')
ax.set_xticks([0,1,2]); ax.set_yticks([0,1,2])
ax.set_xticklabels(['positive','negative','neutral'], rotation=30)
ax.set_yticklabels(['positive','negative','neutral'])
for i in range(3):
    for j in range(3):
        ax.text(j, i, cm[i,j], ha='center', va='center', fontsize=14, fontweight='bold')
ax.set_xlabel('Predicted'); ax.set_ylabel('Actual')
ax.set_title('Cricket Sentiment — Confusion Matrix')
plt.colorbar(im, ax=ax); plt.tight_layout()
plt.savefig('/tmp/sentiment_confusion.png', dpi=150, bbox_inches='tight')
print('Confusion matrix saved: /tmp/sentiment_confusion.png')

# ── Training curves ───────────────────────────────────────────
log_history = trainer.state.log_history
train_loss = [l['loss'] for l in log_history if 'loss' in l and 'eval_loss' not in l]
val_acc    = [l['eval_accuracy'] for l in log_history if 'eval_accuracy' in l]
fig, (ax1,ax2) = plt.subplots(1,2,figsize=(12,4))
ax1.plot(train_loss, label='Train Loss', color='#1f77b4')
ax1.set_title('Training Loss'); ax1.set_xlabel('Log Step'); ax1.legend(); ax1.grid(True,alpha=0.3)
ax2.plot(val_acc,   label='Val Accuracy', color='#2ca02c')
ax2.set_title('Validation Accuracy'); ax2.set_xlabel('Epoch'); ax2.legend(); ax2.grid(True,alpha=0.3)
plt.tight_layout(); plt.savefig('/tmp/sentiment_training.png', dpi=150, bbox_inches='tight')
print('Training curves saved: /tmp/sentiment_training.png')

Step 4 — Testing & Verification

Step 4 exports the model for production deployment and verifies it works correctly in inference mode. A good deployment test is to pass a mix of clearly positive, clearly negative, and clearly neutral commentaries and verify the predictions match expectations. The model is saved in HuggingFace format for direct reloading with from_pretrained() and in pipeline format for the one-line inference API.

Analogy🏏Cricket

🏏 Think of it like cricket: Verifying the network is a batsman's fitness test with three non-negotiable checks. First, form must steadily improve — the loss must fall monotonically over the first 200 epochs, just as a player's error count should drop session after session, not swing wildly. Second, real skill must beat luck — test accuracy above 70% when random guessing is 50%, exactly as a batsman must clearly outscore a tail-ender swinging blindly to prove genuine technique. Third, every part of the technique must be engaged — non-zero gradient norms confirm the learning signal actually flows through all three layers, like checking footwork, backlift, and follow-through are each working rather than one part being frozen. The gold standard is the numerical gradient check: nudge a weight a tiny amount and confirm the loss changes as predicted, just as a coach films from two angles to confirm a fix is real, not imagined. The payoff: these checks prove your from-scratch network is genuinely learning and correctly wired, not just accidentally producing a decent number.

bash

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline as hf_pipeline
import torch

# ── Save and deploy ────────────────────────────────────────────
OUTPUT_PATH = '/tmp/cricket_sentiment_final'
trainer.save_model(OUTPUT_PATH)
tokenizer.save_pretrained(OUTPUT_PATH)
print(f'Model saved to {OUTPUT_PATH}')

# ── Production inference ───────────────────────────────────────
sentiment_pipe = hf_pipeline(
    'text-classification',
    model=OUTPUT_PATH,
    tokenizer=OUTPUT_PATH,
    device=-1   # -1=CPU; 0=GPU
)

# Verification: diverse test inputs
verification_texts = [
    'Rohit Sharma hits a magnificent six to bring up his century',     # expect: positive
    'India lose three wickets in one over — complete collapse',          # expect: negative
    'The toss has been won — India choose to bat first',                 # expect: neutral
    'What a catch by Kohli — unbelievable reflexes at mid-off',         # expect: positive
    'Bumrah is hit for 24 in his last over — expensive spell',           # expect: negative
    'The drinks break is taken at the end of the 15th over',             # expect: neutral
]

print('\n=== Production Inference Verification ===')
results = sentiment_pipe(verification_texts, top_k=1)
for text, result in zip(verification_texts, results):
    label = result[0]['label']
    score = result[0]['score']
    status = '✓' if (
        ('six' in text or 'century' in text or 'catch' in text or 'magnificent' in text) == (label=='positive') or
        ('collapse' in text or 'wicket' in text or 'expensive' in text) == (label=='negative') or
        ('toss' in text or 'break' in text) == (label=='neutral')
    ) else '✗'
    print(f'  {status} [{label:10s} {score:.0%}] {text[:55]}')

Warning: Always test the deployed model with extreme and edge-case inputs before production deployment. Cricket sentiment can be ambiguous: 'India lose the toss but win the match' contains both negative (lose toss) and positive (win match) signals — verify the model's handling of mixed-sentiment commentary matches your production requirements. Add a confidence threshold: only surface high-confidence predictions (score > 0.85) and route low-confidence inputs to a neutral fallback or human review queue.

Extension Challenge: (1) Add a fourth class 'fielding brilliance' (stunning catches, direct-hit run-outs) and observe how the model handles four-class sentiment with limited fielding-specific training data. (2) Implement confidence-threshold routing: use sentiment_pipe() scores to route low-confidence predictions (score < 0.75) to the zero-shot classifier as a fallback, creating a hybrid system that combines fine-tuning accuracy with zero-shot coverage. (3) Evaluate the model on real Cricinfo commentary snippets from a recent IPL match and measure accuracy against manual annotation — the true test of whether the synthetic training data generalises to real commentary style.

Build balanced datasets for multi-class sentiment — imbalanced classes cause the model to default to the majority class for ambiguous inputs, skewing accuracy metrics.
The zero-shot baseline (using facebook/bart-large-mnli) quantifies the value of fine-tuning — if zero-shot achieves 80%+ accuracy, the additional labelling investment for fine-tuning needs explicit justification.
Save models with trainer.save_model(path) AND tokenizer.save_pretrained(path) — both files are needed for full reproduction; missing the tokenizer save breaks downstream loading.
Use pipeline('text-classification', model=path) for production inference — it handles tokenisation, model forward pass, and label mapping in one call with automatic batch processing.
Test deployed models with diverse edge cases including mixed-sentiment commentary before going live — fine-tuned models can be overconfident on out-of-distribution inputs.
Confidence thresholding (route predictions below 0.75 to human review or fallback) is a production best practice — raw model predictions without confidence filtering produce unreliable outputs on ambiguous inputs.