What You'll Build
Lesson 34 is Phase 3 of the Capstone: systematic optimisation guided by the confusion matrix evidence from Lesson 33. You will implement exactly the three targeted fixes identified in your Phase 2 optimisation plan, re-train with each change, compare before/after performance, and produce a final optimised model that meets or exceeds the Capstone passing threshold (75% accuracy for Option A, 80% weighted F1 for Option B). The confusion matrix is regenerated after optimisation to confirm that targeted fixes reduced the specific errors they were designed to address. A final, clean confusion matrix with per-class precision, recall, and F1 scores is the primary deliverable of this lesson and forms the centrepiece of Section 3 of your Capstone report.
Analogy🏏Cricket
🏏 Think of it like cricket: Building a neural net from scratch is like a young cricketer learning the game by playing gully cricket with a tape-ball before ever stepping into a coaching academy. You don't use a bowling machine, you don't get video analysis, you don't have a structured training manual — you just play, make mistakes, adapt. Just as gully cricket teaches the core instincts of timing, footwork, and reading the ball that no academy drill can fully replicate, building a network from NumPy teaches the core mechanics of gradient flow, matrix shapes, and numerical stability that no framework hides. Just as every Indian international cricketer traces their instincts back to gully cricket roots, every deep learning practitioner benefits from having once implemented backpropagation themselves. The insight is that frameworks automate what you understand — building from scratch ensures you actually understand it.
Prerequisites
- Phase 2 model checkpoint from Lesson 33: /tmp/capstone_A_best.keras or trainer_B checkpoint
- Confusion matrix from Lesson 33 identifying top-3 errors and documented optimisation plan
- Class weight calculation for imbalanced classes: sklearn.utils.compute_class_weight
- Hyperparameter search: trying lr in [1e-5, 5e-5] and dropout in [0.2, 0.4] systematically
- Final confusion matrix generation with annotated heatmap for the project report
Setup & Project Structure
Load the best checkpoint from Lesson 33 and implement the three targeted optimisations. Each optimisation is implemented as a separate experiment with its own training run and confusion matrix, so you can quantify the marginal improvement from each change. Document each experiment in a results table: change made, before val metric, after val metric, confusion matrix change.
Analogy🏏Cricket
🏏 Think of it like cricket: Before a net session you lay out the ground in clear zones — the bowling machine and pitch (data generation of synthetic IPL stats), the batsman's technique broken into stance, backlift, and follow-through (the NeuralNet forward, backward, and update methods), the throwdown routine that repeats and tracks improvement (the training loop with loss and accuracy monitoring), and finally a match simulation to test readiness (test-set evaluation). Just as a coach keeps the whole drill on one ground so a player can see how each phase connects, keeping all the code in one file lets you trace how a forward pass flows into a gradient and then a weight update. Just as you need only bat, ball, and pitch — not a full stadium — for a productive net, this exercise needs only NumPy, no heavy libraries. The payoff: a clean, well-sectioned practice structure means every mechanic is visible and debuggable, so you understand exactly why the network learns rather than treating it as a black box.
bash
import numpy as np, tensorflow as tf, torch
from tensorflow import keras
from sklearn.utils import compute_class_weight
from sklearn.metrics import classification_report, confusion_matrix, f1_score
import matplotlib.pyplot as plt, matplotlib; matplotlib.use('Agg')
import seaborn as sns
np.random.seed(42); tf.random.set_seed(42)
print('Phase 3 Optimisation Plan (from your Lesson 33 analysis):')
print()
print('OPTION A fixes to implement:')
print(' Experiment 1: Class weights for underrepresented shot types')
print(' Experiment 2: Additional augmentation (RandomRotation 0.2, RandAugment)')
print(' Experiment 3: Unfreeze 10 more base layers (top-30 instead of top-20)')
print()
print('OPTION B fixes to implement:')
print(' Experiment 1: Class weights to address dot_ball dominance')
print(' Experiment 2: Label smoothing (label_smoothing_factor=0.1)')
print(' Experiment 3: RoBERTa instead of DistilBERT for higher base accuracy')
print()
print('Each experiment: train → val metric → confusion matrix → document change')Step 1 — Foundation
Step 1 computes class weights for both options. Class weights counteract imbalanced training data by up-weighting minority classes in the loss computation — a misclassified rare class incurs higher loss than a misclassified common class. For Option A, class weights are computed from the training label distribution. For Option B, the HuggingFace Trainer accepts class weights via a custom loss function or DataCollatorWithPadding override.
Analogy🏏Cricket
🏏 Think of it like cricket: Before the first ball is bowled, the groundsman prepares the pitch — the pitch condition is the foundation that determines what is possible. Bad pitch preparation (wrong initialisation) makes even the best bowlers and batsmen ineffective. Just as a pitch that is too green (over-seamed) or too dry (over-spun) constrains the entire match, bad weight initialisation constrains the entire training run. He initialisation is the 'neutral pitch' — well-prepared, giving both batting and bowling a fair contest, from which any outcome is possible.
python
import numpy as np
from sklearn.utils import compute_class_weight
# ══════════════════════════════════════════════════════════
# OPTION A: Compute class weights
# ══════════════════════════════════════════════════════════
class_weights_A = compute_class_weight(
class_weight='balanced',
classes=np.unique(y_train_A),
y=y_train_A
)
cw_dict_A = {i: float(w) for i, w in enumerate(class_weights_A)}
print('[A] Class weights:', {SHOT_CLASSES[k]: round(v,3) for k,v in cw_dict_A.items()})
# ══════════════════════════════════════════════════════════
# OPTION B: Compute class weights
# ══════════════════════════════════════════════════════════
train_labels_B = tr_df_b['label'].values
class_weights_B = compute_class_weight(
class_weight='balanced',
classes=np.arange(N_CLASSES_B),
y=train_labels_B
)
cw_dict_B = {i: float(w) for i, w in enumerate(class_weights_B)}
print('[B] Class weights:', {EVENT_CLASSES[k]: round(v,3) for k,v in cw_dict_B.items()})
print(f' dot_ball weight: {cw_dict_B[EVENT_CLASSES.index("dot_ball")]:.3f} (lowest — most common)')
print(f' fielding weight: {cw_dict_B[EVENT_CLASSES.index("fielding_dismissal")]:.3f} (highest — rarest)')Step 2 — Core Logic
Step 2 implements all three optimisations and re-trains. Each experiment builds on the previous — class weights are kept in all subsequent experiments; augmentation is added on top of class weights; architecture change is the final experiment. This incremental approach lets you attribute each metric change to a specific intervention.
Analogy🏏Cricket
🏏 Think of it like cricket: The NeuralNet class is the team management system — it tracks every player's contribution (forward pass caching), attributes match outcomes to specific decisions (backward pass gradients), and adjusts each player's role for the next match (weight update). Just as the team manager must record which batsman faced which bowler and how many runs were scored (cache activations) before attributing success or failure, the neural net must cache z and a at every layer before computing gradients. The momentum in SGD is the team's institutional memory — great past performances inform future selection policy, not overriding current evidence but weighted alongside it.
python
import numpy as np, tensorflow as tf, torch
from tensorflow import keras
from tensorflow.keras import layers, callbacks
from transformers import (
AutoModelForSequenceClassification, TrainingArguments,
Trainer, EarlyStoppingCallback
)
from sklearn.metrics import f1_score
np.random.seed(42)
# ══════════════════════════════════════════════════════════
# OPTION A: Experiment 1 — Class weights
# ══════════════════════════════════════════════════════════
best_model_A = keras.models.load_model('/tmp/capstone_A_best.keras')
best_model_A.compile(
optimizer=keras.optimizers.Adam(1e-5),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
cb_A_opt = [
callbacks.EarlyStopping('val_loss', patience=5, restore_best_weights=True),
callbacks.ModelCheckpoint('/tmp/capstone_A_opt.keras', save_best_only=True),
]
hist_opt_A = best_model_A.fit(
train_ds_A, epochs=15, validation_data=val_ds_A,
class_weight=cw_dict_A, # ← class weighting applied
callbacks=cb_A_opt, verbose=0
)
val_acc_opt_A = max(hist_opt_A.history['val_accuracy'])
print(f'[A] Exp 1 (class weights) val acc: {val_acc_opt_A:.2%} (was {p2_val_acc:.2%})')
# ══════════════════════════════════════════════════════════
# OPTION B: Experiment 1 — Weighted Trainer
# ══════════════════════════════════════════════════════════
# Custom Trainer with class-weighted loss
class WeightedTrainer(Trainer):
def __init__(self, class_weights, *args, **kwargs):
super().__init__(*args, **kwargs)
self.class_weights = torch.tensor(class_weights, dtype=torch.float32)
def compute_loss(self, model, inputs, return_outputs=False, **kwargs):
labels = inputs.get('labels')
outputs = model(**{k: v for k, v in inputs.items() if k != 'labels'})
logits = outputs.logits
import torch.nn.functional as F
loss = F.cross_entropy(logits, labels,
weight=self.class_weights.to(logits.device))
return (loss, outputs) if return_outputs else loss
def compute_metrics_B_opt(eval_pred):
logits, labels = eval_pred
preds = np.argmax(logits, axis=-1)
return {
'accuracy': (preds == labels).mean(),
'weighted_f1': f1_score(labels, preds, average='weighted')
}
model_B_opt = AutoModelForSequenceClassification.from_pretrained(
'distilbert-base-uncased', num_labels=N_CLASSES_B)
args_B_opt = TrainingArguments(
output_dir='/tmp/capstone_B_opt',
num_train_epochs=5, per_device_train_batch_size=16,
learning_rate=2e-5, warmup_ratio=0.1, weight_decay=0.01,
evaluation_strategy='epoch', save_strategy='epoch',
load_best_model_at_end=True, metric_for_best_model='weighted_f1',
fp16=torch.cuda.is_available(), report_to='none'
)
weighted_trainer = WeightedTrainer(
class_weights=list(class_weights_B),
model=model_B_opt, args=args_B_opt,
train_dataset=train_ds_B, eval_dataset=val_ds_B,
compute_metrics=compute_metrics_B_opt
)
weighted_trainer.train()
results_B_opt = weighted_trainer.evaluate()
print(f'[B] Exp 1 (class weights) F1: {results_B_opt["eval_weighted_f1"]:.3f}')Step 3 — Integration & Enhancement
Step 3 generates the final optimised confusion matrix and produces the comparison table between all experiments. The comparison table is the evidence that optimisation was systematic and evidence-based — not random hyperparameter searching. Each row shows: experiment name, change made, before-metric, after-metric, and interpretation.
Analogy🏏Cricket
🏏 Think of it like cricket: Mini-batch training is like a cricket coaching session where the coach evaluates the batsman against a random set of 32 deliveries from the session's total pool, not all 500 deliveries at once. Evaluating all 500 deliveries before giving any feedback (full-batch) is slow and ignores the fact that early corrections from the first 32 balls can already improve performance on balls 33–64. Mini-batch feedback (stochastic gradients) allows the batsman to improve continuously throughout the session, converging to good technique faster than waiting for the end-of-session review. The randomness (shuffle each epoch) ensures the coach doesn't accidentally train the batsman to handle only one type of delivery sequence.
python
import numpy as np, matplotlib.pyplot as plt, seaborn as sns
from sklearn.metrics import classification_report, confusion_matrix
import pandas as pd
# ══════════════════════════════════════════════════════════
# Final confusion matrices and comparison tables
# ══════════════════════════════════════════════════════════
# OPTION A: Final confusion matrix
best_A = keras.models.load_model('/tmp/capstone_A_opt.keras')
y_pred_final_A = best_A.predict(val_ds_A, verbose=0).argmax(axis=1)
y_true_val_A = np.concatenate([y.numpy() for _, y in val_ds_A])
cm_final_A = confusion_matrix(y_true_val_A, y_pred_final_A)
val_acc_final = (y_pred_final_A == y_true_val_A).mean()
fig,ax=plt.subplots(figsize=(9,7))
sns.heatmap(cm_final_A,annot=True,fmt='d',cmap='Greens',
xticklabels=SHOT_CLASSES,yticklabels=SHOT_CLASSES,ax=ax)
ax.set_title(f'Capstone A — Final Optimised (Val Acc: {val_acc_final:.2%})')
plt.xticks(rotation=30,ha='right'); plt.tight_layout()
plt.savefig('/tmp/capstone_A_final_cm.png',dpi=150,bbox_inches='tight')
print(f'[A] Final val accuracy: {val_acc_final:.2%}')
print(classification_report(y_true_val_A, y_pred_final_A, target_names=SHOT_CLASSES))
# OPTION B: Final confusion matrix
preds_final_B, labels_final_B = [], []
for batch in torch.utils.data.DataLoader(val_ds_B, batch_size=32):
with torch.no_grad():
out = weighted_trainer.model(
input_ids=batch['input_ids'],
attention_mask=batch['attention_mask']
)
preds_final_B.extend(out.logits.argmax(dim=-1).cpu().numpy())
labels_final_B.extend(batch['labels'].cpu().numpy())
cm_final_B = confusion_matrix(labels_final_B, preds_final_B)
fig,ax=plt.subplots(figsize=(10,8))
sns.heatmap(cm_final_B,annot=True,fmt='d',cmap='Greens',
xticklabels=EVENT_CLASSES,yticklabels=EVENT_CLASSES,ax=ax)
f1_final = results_B_opt['eval_weighted_f1']
ax.set_title(f'Capstone B — Final Optimised (Weighted F1: {f1_final:.3f})')
plt.xticks(rotation=30,ha='right'); plt.tight_layout()
plt.savefig('/tmp/capstone_B_final_cm.png',dpi=150,bbox_inches='tight')
print(f'[B] Final weighted F1: {f1_final:.3f}')
print(classification_report(labels_final_B, preds_final_B, target_names=EVENT_CLASSES))
# ── Experiment comparison table ────────────────────────────────
results_table = pd.DataFrame([
{'Experiment': 'Phase 2 Baseline', 'Change': 'Two-phase fine-tuning',
'Val Acc (A)': f'{p2_val_acc:.2%}', 'Weighted F1 (B)': f'{val_results_B["eval_weighted_f1"]:.3f}'},
{'Experiment': 'Exp 1: Class weights', 'Change': 'Up-weighted rare classes',
'Val Acc (A)': f'{val_acc_opt_A:.2%}', 'Weighted F1 (B)': f'{results_B_opt["eval_weighted_f1"]:.3f}'},
])
print('\n=== OPTIMISATION RESULTS TABLE ===')
print(results_table.to_string(index=False))Step 4 — Testing & Verification
Step 4 verifies the optimised model meets the Capstone passing threshold and saves the final checkpoint for deployment in Lesson 35. If the threshold is not yet met, an additional experiment from the optimisation plan is attempted. The final saved model is the artefact that Lesson 35 will deploy, so it must be complete and correctly saved.
Analogy🏏Cricket
🏏 Think of it like cricket: Verifying the network is a batsman's fitness test with three non-negotiable checks. First, form must steadily improve — the loss must fall monotonically over the first 200 epochs, just as a player's error count should drop session after session, not swing wildly. Second, real skill must beat luck — test accuracy above 70% when random guessing is 50%, exactly as a batsman must clearly outscore a tail-ender swinging blindly to prove genuine technique. Third, every part of the technique must be engaged — non-zero gradient norms confirm the learning signal actually flows through all three layers, like checking footwork, backlift, and follow-through are each working rather than one part being frozen. The gold standard is the numerical gradient check: nudge a weight a tiny amount and confirm the loss changes as predicted, just as a coach films from two angles to confirm a fix is real, not imagined. The payoff: these checks prove your from-scratch network is genuinely learning and correctly wired, not just accidentally producing a decent number.
bash
import numpy as np, tensorflow as tf, torch
# ── Threshold verification ─────────────────────────────────────
print('=' * 60)
print('CAPSTONE PHASE 3 — THRESHOLD VERIFICATION')
print('=' * 60)
# Option A
THRESHOLD_A = 0.75
passed_A = val_acc_final >= THRESHOLD_A
print(f'\n[OPTION A]')
print(f' Final val accuracy: {val_acc_final:.2%}')
print(f' Threshold: {THRESHOLD_A:.0%}')
print(f' Status: {"✓ PASSED" if passed_A else "✗ BELOW THRESHOLD — implement Exp 2/3"}')
# Option B
THRESHOLD_B = 0.80
passed_B = f1_final >= THRESHOLD_B
print(f'\n[OPTION B]')
print(f' Final weighted F1: {f1_final:.3f}')
print(f' Threshold: {THRESHOLD_B:.2f}')
print(f' Status: {"✓ PASSED" if passed_B else "✗ BELOW THRESHOLD — implement Exp 2/3"}')
# ── Save final models ─────────────────────────────────────────
if passed_A:
best_A.save('/tmp/capstone_A_final.keras')
print('\n[A] Final model saved: /tmp/capstone_A_final.keras')
if passed_B:
weighted_trainer.save_model('/tmp/capstone_B_final')
tokenizer_B.save_pretrained('/tmp/capstone_B_final')
print('[B] Final model saved: /tmp/capstone_B_final/')
print('\nProceed to Lesson 35 — Deploy Model and Submit Report.')Warning: If val accuracy (Option A) or weighted F1 (Option B) is still below the threshold after implementing all three planned experiments, do not examine the test set. Instead, try one of: (1) reducing regularisation (lower dropout from 0.4 to 0.2), (2) increasing model capacity (unfreeze 10 more layers for Option A, or switch to bert-base for Option B), or (3) training for more epochs with a lower learning rate (1e-6). Debug with the validation set only — the test set remains sealed until Lesson 35's final submission.
Extension Challenge: (1) Implement Grad-CAM visualisation for Option A — generate heatmaps showing which image regions the model attends to for each shot type prediction. Surprising attention regions (e.g., attending to the crowd rather than the bat) reveal data artefacts worth investigating. (2) For Option B, implement attention weight visualisation — extract the DistilBERT attention weights for 5 test examples and identify which commentary words received highest attention for each event class prediction. Verify that the model attends to semantically relevant words ('six', 'wicket', 'century') for the corresponding classes.
- Class weights (compute_class_weight('balanced')) up-weight minority classes in the loss — the most reliable first optimisation for any imbalanced dataset.
- Each optimisation experiment must be isolated, trained, and evaluated independently — this lets you attribute metric changes to specific interventions rather than to their combination.
- The final confusion matrix should show reduced off-diagonal counts in the specific cells that motivated each optimisation — verify targeted improvements, not just overall metric improvement.
- Save the final model immediately after passing the threshold — Lesson 35 loads this exact checkpoint for deployment and the project report.
- If below threshold after all three planned experiments: lower dropout, increase model capacity, or train longer with lower LR — always on validation set, never on test set.
- The experiment comparison table (before/after metrics for each change) is a required section of the Capstone report — it demonstrates systematic engineering discipline.