What You'll Build
Lesson 33 is Phase 2 of the Capstone: designing and training the model on the verified data pipeline from Lesson 32. You will write a documented architecture justification (why this architecture for this task), implement the training loop with the full callback suite, track training and validation metrics across epochs, and conduct the first systematic evaluation with per-class metrics and training curves. The model does not need to be perfect after Lesson 33 — the purpose of this phase is to establish a working baseline with clean training behaviour. Lesson 34 is dedicated to optimisation. The key deliverable is a training run that converges without NaN loss, shows a decreasing validation loss trend, and produces a confusion matrix that reveals specific weaknesses to address in the next phase.
Analogy🏏Cricket
🏏 Think of it like cricket: Building a neural net from scratch is like a young cricketer learning the game by playing gully cricket with a tape-ball before ever stepping into a coaching academy. You don't use a bowling machine, you don't get video analysis, you don't have a structured training manual — you just play, make mistakes, adapt. Just as gully cricket teaches the core instincts of timing, footwork, and reading the ball that no academy drill can fully replicate, building a network from NumPy teaches the core mechanics of gradient flow, matrix shapes, and numerical stability that no framework hides. Just as every Indian international cricketer traces their instincts back to gully cricket roots, every deep learning practitioner benefits from having once implemented backpropagation themselves. The insight is that frameworks automate what you understand — building from scratch ensures you actually understand it.
Prerequisites
- Completed Lesson 32 pipeline: train_ds_A/train_ds_B, val_ds_A/val_ds_B with verified shapes and class distributions
- EfficientNetB0 two-phase training: frozen base (Phase 1) → unfreeze top 20 layers (Phase 2) for Option A
- DistilBERT fine-tuning recipe: lr=2e-5, warmup_ratio=0.1, weight_decay=0.01 for Option B
- EarlyStopping + ModelCheckpoint callbacks for both options
- per-class metrics: classification_report, confusion_matrix from sklearn
Setup & Project Structure
This lesson uses the data pipelines from Lesson 32. Start by loading the verified datasets and confirming the sanity checks still pass. Then build the model, write the architecture justification, and run the training loop with full monitoring. The architecture justification section is mandatory for the Capstone report — it demonstrates that architecture decisions were made deliberately rather than by copying a template.
Analogy🏏Cricket
🏏 Think of it like cricket: Before a net session you lay out the ground in clear zones — the bowling machine and pitch (data generation of synthetic IPL stats), the batsman's technique broken into stance, backlift, and follow-through (the NeuralNet forward, backward, and update methods), the throwdown routine that repeats and tracks improvement (the training loop with loss and accuracy monitoring), and finally a match simulation to test readiness (test-set evaluation). Just as a coach keeps the whole drill on one ground so a player can see how each phase connects, keeping all the code in one file lets you trace how a forward pass flows into a gradient and then a weight update. Just as you need only bat, ball, and pitch — not a full stadium — for a productive net, this exercise needs only NumPy, no heavy libraries. The payoff: a clean, well-sectioned practice structure means every mechanic is visible and debuggable, so you understand exactly why the network learns rather than treating it as a black box.
bash
# Assume all imports and data from Lesson 32 are available
# Reload if running in a new session:
import numpy as np, tensorflow as tf, torch
from tensorflow import keras
from tensorflow.keras import layers, callbacks
from transformers import (
AutoModelForSequenceClassification, TrainingArguments,
Trainer, EarlyStoppingCallback
)
from sklearn.metrics import classification_report, confusion_matrix
import matplotlib.pyplot as plt, matplotlib; matplotlib.use('Agg')
np.random.seed(42); tf.random.set_seed(42); torch.manual_seed(42)
print('Architecture Justification Template (complete this in your report):')
print()
print('OPTION A JUSTIFICATION:')
print(' Architecture: EfficientNetB0 pretrained on ImageNet')
print(' Reason: [YOUR TEXT] e.g., "EfficientNetB0 provides the best accuracy/')
print(' parameter ratio at 5.3M params. ImageNet pretraining gives')
print(' edge/texture features that transfer to cricket shot images."')
print(' Alternative considered: ResNet50 — rejected because 25M params risks')
print(' overfitting on our 840-image training set.')
print()
print('OPTION B JUSTIFICATION:')
print(' Architecture: DistilBERT-base-uncased fine-tuned')
print(' Reason: [YOUR TEXT] e.g., "DistilBERT has 66M params vs BERT-base 110M,')
print(' achieving 97% of BERT performance at 60% size. Cricket')
print(' commentary is short (<50 words), so the 512-token context')
print(' limit is never a constraint. Bidirectional attention captures')
print(' shot context better than GPT-2 causal attention."')
print(' Alternative considered: Training from scratch — rejected because we')
print(' have only 530 training texts, insufficient for learning')
print(' language representations from random initialisation.')Step 1 — Foundation
Step 1 builds and compiles the model with the justified architecture. For Option A, this is the two-phase EfficientNetB0 setup with frozen base and trainable head. For Option B, this is DistilBERT loaded with the classification head. Both models are verified for parameter count, trainable vs frozen parameter split, and output shape.
Analogy🏏Cricket
🏏 Think of it like cricket: Before the first ball is bowled, the groundsman prepares the pitch — the pitch condition is the foundation that determines what is possible. Bad pitch preparation (wrong initialisation) makes even the best bowlers and batsmen ineffective. Just as a pitch that is too green (over-seamed) or too dry (over-spun) constrains the entire match, bad weight initialisation constrains the entire training run. He initialisation is the 'neutral pitch' — well-prepared, giving both batting and bowling a fair contest, from which any outcome is possible.
python
import tensorflow as tf, torch, numpy as np
from tensorflow import keras
from tensorflow.keras import layers
from transformers import AutoModelForSequenceClassification
# ══════════════════════════════════════════════════════════
# OPTION A: Build EfficientNetB0 model
# ══════════════════════════════════════════════════════════
N_CLASSES_A = 6
SHOT_CLASSES = ['cover_drive','pull_shot','sweep','flick','straight_drive','helicopter']
def build_phase1_A():
"""Phase 1: frozen base + trainable head."""
base = keras.applications.EfficientNetB0(
include_top=False, weights=None, # use 'imagenet' with real data
input_shape=(224,224,3), pooling='avg')
base.trainable = False
inp = keras.Input((224,224,3))
x = layers.RandomFlip('horizontal')(inp) # in-graph augmentation
x = layers.RandomRotation(0.1)(x)
x = layers.RandomZoom(0.1)(x)
x = layers.RandomBrightness(0.2)(x)
x = keras.applications.efficientnet.preprocess_input(x)
x = base(x, training=False)
x = layers.Dense(256, activation='relu')(x)
x = layers.Dropout(0.4)(x)
x = layers.Dense(N_CLASSES_A, activation='softmax', dtype='float32')(x)
m = keras.Model(inp, x)
m.compile(
optimizer=keras.optimizers.Adam(1e-3),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
return m, base
model_A, base_A = build_phase1_A()
train_params_A = sum(v.numpy().size for v in model_A.trainable_variables)
total_params_A = model_A.count_params()
print(f'[A] Phase 1 — Trainable: {train_params_A:,} / {total_params_A:,} ({train_params_A/total_params_A:.1%})')
# ══════════════════════════════════════════════════════════
# OPTION B: Load DistilBERT
# ══════════════════════════════════════════════════════════
N_CLASSES_B = 7
EVENT_CLASSES = ['batting_milestone','bowling_wicket','fielding_dismissal',
'six_scored','four_scored','dot_ball','match_admin']
model_B = AutoModelForSequenceClassification.from_pretrained(
'distilbert-base-uncased',
num_labels=N_CLASSES_B,
id2label={i:c for i,c in enumerate(EVENT_CLASSES)},
label2id={c:i for i,c in enumerate(EVENT_CLASSES)}
)
total_B = sum(p.numel() for p in model_B.parameters())
train_B = sum(p.numel() for p in model_B.parameters() if p.requires_grad)
print(f'[B] DistilBERT — Trainable: {train_B:,} / {total_B:,} ({train_B/total_B:.1%})')Step 2 — Core Logic
Step 2 runs the full training loop for both phases (Option A) or the complete fine-tuning run (Option B) with the production-grade callback suite. After training, the validation metrics are inspected and the preliminary confusion matrix is generated. This confusion matrix guides the optimisation strategy in Lesson 34.
Analogy🏏Cricket
🏏 Think of it like cricket: The NeuralNet class is the team management system — it tracks every player's contribution (forward pass caching), attributes match outcomes to specific decisions (backward pass gradients), and adjusts each player's role for the next match (weight update). Just as the team manager must record which batsman faced which bowler and how many runs were scored (cache activations) before attributing success or failure, the neural net must cache z and a at every layer before computing gradients. The momentum in SGD is the team's institutional memory — great past performances inform future selection policy, not overriding current evidence but weighted alongside it.
python
import numpy as np, tensorflow as tf, torch
from tensorflow import keras
from tensorflow.keras import callbacks
from transformers import TrainingArguments, Trainer, EarlyStoppingCallback
from sklearn.metrics import classification_report, confusion_matrix
np.random.seed(42)
# ══════════════════════════════════════════════════════════
# OPTION A: Two-phase training
# ══════════════════════════════════════════════════════════
cb_A = [
callbacks.EarlyStopping('val_loss', patience=7, restore_best_weights=True, verbose=1),
callbacks.ModelCheckpoint('/tmp/capstone_A_best.keras', save_best_only=True, verbose=0),
callbacks.ReduceLROnPlateau('val_loss', factor=0.5, patience=4, min_lr=1e-6, verbose=1)
]
# Phase 1: frozen base
print('[A] Phase 1 Training (frozen base)...')
hist1_A = model_A.fit(train_ds_A, epochs=20, validation_data=val_ds_A, callbacks=cb_A, verbose=1)
p1_val_acc = max(hist1_A.history['val_accuracy'])
print(f'Phase 1 best val accuracy: {p1_val_acc:.2%}')
# Phase 2: unfreeze top 20 layers
base_A.trainable = True
for layer in base_A.layers[:-20]:
layer.trainable = False
model_A.compile(
optimizer=keras.optimizers.Adam(1e-5),
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)
print('\n[A] Phase 2 Training (top-20 layers fine-tuned)...')
hist2_A = model_A.fit(train_ds_A, epochs=15, validation_data=val_ds_A, callbacks=cb_A, verbose=1)
p2_val_acc = max(hist2_A.history['val_accuracy'])
print(f'Phase 2 best val accuracy: {p2_val_acc:.2%}')
# ══════════════════════════════════════════════════════════
# OPTION B: DistilBERT fine-tuning
# ══════════════════════════════════════════════════════════
def compute_metrics_B(eval_pred):
from sklearn.metrics import f1_score
logits, labels = eval_pred
preds = np.argmax(logits, axis=-1)
acc = (preds == labels).mean()
f1_w = f1_score(labels, preds, average='weighted')
return {'accuracy': acc, 'weighted_f1': f1_w}
args_B = TrainingArguments(
output_dir='/tmp/capstone_B',
num_train_epochs=5,
per_device_train_batch_size=16,
per_device_eval_batch_size=32,
learning_rate=2e-5,
warmup_ratio=0.1,
weight_decay=0.01,
evaluation_strategy='epoch',
save_strategy='epoch',
load_best_model_at_end=True,
metric_for_best_model='weighted_f1',
fp16=torch.cuda.is_available(),
logging_steps=20,
report_to='none'
)
trainer_B = Trainer(
model=model_B, args=args_B,
train_dataset=train_ds_B, eval_dataset=val_ds_B,
compute_metrics=compute_metrics_B,
callbacks=[EarlyStoppingCallback(early_stopping_patience=2)]
)
print('[B] DistilBERT fine-tuning...')
trainer_B.train()
val_results_B = trainer_B.evaluate()
print(f'Val accuracy: {val_results_B["eval_accuracy"]:.2%}')
print(f'Val weighted F1: {val_results_B["eval_weighted_f1"]:.3f}')Step 3 — Integration & Enhancement
Step 3 generates the training curves, preliminary confusion matrix, and per-class classification report. These three outputs are the core of the Phase 2 evaluation and directly feed the optimisation strategy in Lesson 34. The confusion matrix should be inspected carefully: off-diagonal cells with high counts identify the model's systematic errors, which are almost always more fixable than random errors.
Analogy🏏Cricket
🏏 Think of it like cricket: Mini-batch training is like a cricket coaching session where the coach evaluates the batsman against a random set of 32 deliveries from the session's total pool, not all 500 deliveries at once. Evaluating all 500 deliveries before giving any feedback (full-batch) is slow and ignores the fact that early corrections from the first 32 balls can already improve performance on balls 33–64. Mini-batch feedback (stochastic gradients) allows the batsman to improve continuously throughout the session, converging to good technique faster than waiting for the end-of-session review. The randomness (shuffle each epoch) ensures the coach doesn't accidentally train the batsman to handle only one type of delivery sequence.
python
import numpy as np, matplotlib.pyplot as plt
from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
# ══════════════════════════════════════════════════════════
# OPTION A: Evaluate and visualise
# ══════════════════════════════════════════════════════════
# Training curves (combined phases)
def plot_combined_A(h1, h2, path='/tmp/capstone_A_curves.png'):
fig,(ax1,ax2)=plt.subplots(1,2,figsize=(14,5))
e1=range(1,len(h1.history['loss'])+1)
e2=range(len(e1)+1,len(e1)+len(h2.history['loss'])+1)
for ax, key, title in [(ax1,'loss','Loss'),(ax2,'accuracy','Accuracy')]:
ax.plot(e1,h1.history[key],'b-',label='Ph1 Train')
ax.plot(e1,h1.history[f'val_{key}'],'b--',label='Ph1 Val')
ax.plot(e2,h2.history[key],'r-',label='Ph2 Train')
ax.plot(e2,h2.history[f'val_{key}'],'r--',label='Ph2 Val')
ax.axvline(len(e1),color='gray',linestyle=':',label='Fine-tune start')
ax.set_title(f'Capstone A — {title}'); ax.legend(); ax.grid(True,alpha=0.3)
plt.tight_layout(); plt.savefig(path,dpi=150,bbox_inches='tight')
print(f'Saved: {path}')
plot_combined_A(hist1_A, hist2_A)
# Predictions on validation set
y_pred_A = model_A.predict(val_ds_A, verbose=0).argmax(axis=1)
y_true_A = np.concatenate([y.numpy() for _, y in val_ds_A])
print('[A] Preliminary Per-Class Report:')
print(classification_report(y_true_A, y_pred_A, target_names=SHOT_CLASSES))
# Confusion matrix
cm_A = confusion_matrix(y_true_A, y_pred_A)
fig,ax=plt.subplots(figsize=(8,7))
sns.heatmap(cm_A,annot=True,fmt='d',cmap='Blues',
xticklabels=SHOT_CLASSES,yticklabels=SHOT_CLASSES,ax=ax)
ax.set_xlabel('Predicted'); ax.set_ylabel('Actual')
ax.set_title('Capstone A — Confusion Matrix (Phase 2 Val)')
plt.xticks(rotation=30,ha='right'); plt.tight_layout()
plt.savefig('/tmp/capstone_A_confusion.png',dpi=150,bbox_inches='tight')
print('Confusion matrix saved.')
# ══════════════════════════════════════════════════════════
# OPTION B: Evaluate and visualise
# ══════════════════════════════════════════════════════════
# Detailed per-class metrics
preds_B, labels_B = [], []
for batch in torch.utils.data.DataLoader(val_ds_B, batch_size=32):
with torch.no_grad():
out = trainer_B.model(
input_ids=batch['input_ids'],
attention_mask=batch['attention_mask']
)
preds_B.extend(out.logits.argmax(dim=-1).cpu().numpy())
labels_B.extend(batch['labels'].cpu().numpy())
print('[B] Preliminary Per-Class Report:')
print(classification_report(labels_B, preds_B, target_names=EVENT_CLASSES))
cm_B = confusion_matrix(labels_B, preds_B)
fig,ax=plt.subplots(figsize=(9,8))
sns.heatmap(cm_B,annot=True,fmt='d',cmap='Blues',
xticklabels=EVENT_CLASSES,yticklabels=EVENT_CLASSES,ax=ax)
ax.set_xlabel('Predicted'); ax.set_ylabel('Actual')
ax.set_title('Capstone B — Confusion Matrix (Val)')
plt.xticks(rotation=30,ha='right'); plt.tight_layout()
plt.savefig('/tmp/capstone_B_confusion.png',dpi=150,bbox_inches='tight')
print('Confusion matrix saved.')Step 4 — Testing & Verification
Step 4 confirms the Phase 2 deliverables: a training run that converged cleanly, a preliminary confusion matrix that identifies the model's weaknesses, and a written optimisation plan for Lesson 34. The optimisation plan is a 3-bullet list identifying the top-3 errors from the confusion matrix and the specific changes planned to address each.
Analogy🏏Cricket
🏏 Think of it like cricket: Verifying the network is a batsman's fitness test with three non-negotiable checks. First, form must steadily improve — the loss must fall monotonically over the first 200 epochs, just as a player's error count should drop session after session, not swing wildly. Second, real skill must beat luck — test accuracy above 70% when random guessing is 50%, exactly as a batsman must clearly outscore a tail-ender swinging blindly to prove genuine technique. Third, every part of the technique must be engaged — non-zero gradient norms confirm the learning signal actually flows through all three layers, like checking footwork, backlift, and follow-through are each working rather than one part being frozen. The gold standard is the numerical gradient check: nudge a weight a tiny amount and confirm the loss changes as predicted, just as a coach films from two angles to confirm a fix is real, not imagined. The payoff: these checks prove your from-scratch network is genuinely learning and correctly wired, not just accidentally producing a decent number.
bash
# Phase 2 deliverables checklist
print('=' * 60)
print('CAPSTONE PHASE 2 — DELIVERABLES CHECKLIST')
print('=' * 60)
print('\n[OPTION A]')
print(f' ✓ Phase 1 val accuracy: {p1_val_acc:.2%}')
print(f' ✓ Phase 2 val accuracy: {p2_val_acc:.2%}')
print(f' ✓ Training curves saved: /tmp/capstone_A_curves.png')
print(f' ✓ Confusion matrix saved: /tmp/capstone_A_confusion.png')
print()
print(' Optimisation Plan (complete based on your confusion matrix):')
print(' 1. Most confused pair: [class X] → [class Y]')
print(' Plan: Add more augmentation for class X to improve distinction')
print(' 2. Lowest recall class: [class Z]')
print(' Plan: Check class Z has sufficient training examples (balance)')
print(' 3. Overall val acc below 75%:')
print(' Plan: Unfreeze additional 10 base layers, reduce LR to 5e-6')
print('\n[OPTION B]')
val_f1 = val_results_B.get('eval_weighted_f1', 0)
print(f' ✓ Val accuracy: {val_results_B["eval_accuracy"]:.2%}')
print(f' ✓ Val weighted F1: {val_f1:.3f}')
print(f' ✓ Confusion matrix saved: /tmp/capstone_B_confusion.png')
print()
print(' Optimisation Plan (complete based on your confusion matrix):')
print(' 1. dot_ball recall likely high, rare class recall likely low')
print(' Plan: Add class_weight to Trainer to up-weight rare classes')
print(' 2. batting_milestone vs six_scored confusion (both positive events)')
print(' Plan: Add more diverse training examples for each class')
print(' 3. Weighted F1 below 0.80:')
print(' Plan: Try RoBERTa instead of DistilBERT for higher base accuracy')
print('\nProceed to Lesson 34 — Optimise and Generate Confusion Matrix.')Warning: If training loss goes to NaN in the first epoch for Option A (image), the learning rate is too high for the current model state. The most common cause is starting Phase 2 fine-tuning without compiling the model with the new lower learning rate — if model.compile() is not called after base_A.trainable = True, the optimiser retains the Phase 1 learning rate (1e-3) which is 100× too large for fine-tuning pretrained layers. Always compile after changing trainable status.
Extension Challenge: (1) Implement gradient checkpointing for Option A to reduce GPU memory usage by 40% — enable with model.gradient_checkpointing_enable() to allow training larger batch sizes. (2) For Option B, implement layer-wise learning rate decay (LLRD): assign lr × 0.9^(n_layers - layer_idx) to each DistilBERT layer using parameter groups in the optimiser, applying lower rates to earlier layers that encode more general features.
- Write the architecture justification before building the model — explain why this architecture was chosen over alternatives, citing parameter count, pre-training quality, and dataset size compatibility.
- Phase 1 (frozen base) must converge before Phase 2 (fine-tuning) begins — if Phase 1 never improves, the data pipeline has a bug, not the architecture.
- Always recompile after changing trainable status in Keras — the optimiser must be rebuilt to correctly handle the new set of trainable parameters and their learning rates.
- Use compute_metrics to return both accuracy AND weighted_f1 for Option B — tracking both metrics reveals when a model improves accuracy by over-predicting majority classes at the expense of rare class F1.
- The confusion matrix is the most valuable output of Phase 2 — inspect off-diagonal cells to identify systematic errors and plan targeted remediation in Lesson 34.
- The Phase 2 optimisation plan is a contract with yourself: write 3 specific changes you will make in Lesson 34 based on the confusion matrix evidence — not general improvements, specific targeted fixes.