What You'll Build
Lesson 32 is Phase 1 of the Capstone: building a complete, production-quality data pipeline for your chosen project. You will generate or load your dataset, construct preprocessing and augmentation pipelines, implement correct train/validation/test splits with no data leakage, and build efficient tf.data or HuggingFace Dataset loaders that are ready for model training in Lesson 33. The data pipeline is the most underestimated component of any ML project — it determines the quality ceiling of every downstream experiment. A buggy data pipeline can make a great model appear useless, or make a mediocre model appear excellent. By the end of this lesson, you will have a verified, leakage-free data pipeline with documented statistics (class distribution, split sizes, augmentation visual inspection) that forms the bedrock of your Capstone submission.
Analogy🏏Cricket
🏏 Think of it like cricket: Building a neural net from scratch is like a young cricketer learning the game by playing gully cricket with a tape-ball before ever stepping into a coaching academy. You don't use a bowling machine, you don't get video analysis, you don't have a structured training manual — you just play, make mistakes, adapt. Just as gully cricket teaches the core instincts of timing, footwork, and reading the ball that no academy drill can fully replicate, building a network from NumPy teaches the core mechanics of gradient flow, matrix shapes, and numerical stability that no framework hides. Just as every Indian international cricketer traces their instincts back to gully cricket roots, every deep learning practitioner benefits from having once implemented backpropagation themselves. The insight is that frameworks automate what you understand — building from scratch ensures you actually understand it.
Prerequisites
- train_test_split with stratify= for balanced class distribution across splits
- MinMaxScaler or StandardScaler fitted ONLY on training data — transform val/test with the same fitted object
- Augmentation layers (RandomFlip, RandomRotation) inside the model graph for Option A
- HuggingFace Dataset.from_pandas().map(tokenize_fn) for Option B tokenisation
- tf.data pipeline: .shuffle().batch().prefetch(AUTOTUNE) for GPU-efficient loading
Setup & Project Structure
Both pipeline paths (image and text) follow the same five-step structure: generate or load data, inspect class distribution, split with stratification, apply preprocessing (fit on train only), and build loaders with verification checks. The verification step — printing split sizes, class distributions, and sample previews — is not optional. It is the only way to catch data leakage, label encoding errors, or augmentation bugs before they corrupt training in Lesson 33.
Analogy🏏Cricket
🏏 Think of it like cricket: Before a net session you lay out the ground in clear zones — the bowling machine and pitch (data generation of synthetic IPL stats), the batsman's technique broken into stance, backlift, and follow-through (the NeuralNet forward, backward, and update methods), the throwdown routine that repeats and tracks improvement (the training loop with loss and accuracy monitoring), and finally a match simulation to test readiness (test-set evaluation). Just as a coach keeps the whole drill on one ground so a player can see how each phase connects, keeping all the code in one file lets you trace how a forward pass flows into a gradient and then a weight update. Just as you need only bat, ball, and pitch — not a full stadium — for a productive net, this exercise needs only NumPy, no heavy libraries. The payoff: a clean, well-sectioned practice structure means every mechanic is visible and debuggable, so you understand exactly why the network learns rather than treating it as a black box.
bash
# Setup for both options
import numpy as np, pandas as pd, tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from transformers import AutoTokenizer
from datasets import Dataset
import torch
np.random.seed(42); tf.random.set_seed(42); torch.manual_seed(42)
# Project structure:
# capstone/
# ├── data/
# │ ├── raw/ raw images or commentary texts
# │ ├── processed/ normalised arrays or HF datasets
# │ └── splits/ train/val/test indices
# ├── pipeline_A.py image pipeline (Option A)
# ├── pipeline_B.py text pipeline (Option B)
# └── verify_pipeline.py data quality checks
print('Capstone project structure ready')Step 1 — Foundation
Step 1 generates the dataset and verifies class balance. Both options require balanced class distributions — imbalanced classes require additional handling (class weights, oversampling, or weighted F1 metrics) that should be documented in the project report. A class distribution histogram is the first figure in any credible ML project report.
Analogy🏏Cricket
🏏 Think of it like cricket: Before the first ball is bowled, the groundsman prepares the pitch — the pitch condition is the foundation that determines what is possible. Bad pitch preparation (wrong initialisation) makes even the best bowlers and batsmen ineffective. Just as a pitch that is too green (over-seamed) or too dry (over-spun) constrains the entire match, bad weight initialisation constrains the entire training run. He initialisation is the 'neutral pitch' — well-prepared, giving both batting and bowling a fair contest, from which any outcome is possible.
python
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
np.random.seed(42)
# ══════════════════════════════════════════════════════════
# OPTION A: Image Dataset Pipeline
# ══════════════════════════════════════════════════════════
SHOT_CLASSES_A = ['cover_drive','pull_shot','sweep','flick','straight_drive','helicopter']
N_PER_CLASS_A = 200 # 1200 total synthetic images
IMG_SIZE_A = 224
def generate_synthetic_images(n_per_class, img_size, n_classes):
"""Synthetic 224×224×3 images with class-specific colour bias."""
X, y = [], []
for cls in range(n_classes):
imgs = np.random.randint(0, 256, (n_per_class, img_size, img_size, 3), dtype=np.uint8)
# Add class-specific colour signature (simulates real visual differences)
imgs[:, :, :, cls % 3] = np.clip(imgs[:, :, :, cls % 3] + 40, 0, 255)
X.append(imgs); y.extend([cls] * n_per_class)
return np.concatenate(X), np.array(y)
X_img, y_img = generate_synthetic_images(N_PER_CLASS_A, IMG_SIZE_A, len(SHOT_CLASSES_A))
print(f'[A] Image dataset: X={X_img.shape}, y={y_img.shape}')
# ══════════════════════════════════════════════════════════
# OPTION B: Text Dataset Pipeline
# ══════════════════════════════════════════════════════════
EVENT_CLASSES_B = ['batting_milestone','bowling_wicket','fielding_dismissal',
'six_scored','four_scored','dot_ball','match_admin']
# Realistic imbalance: dot_ball >> batting_milestone
N_PER_CLASS_B = {'batting_milestone':60,'bowling_wicket':80,'fielding_dismissal':40,
'six_scored':100,'four_scored':120,'dot_ball':300,'match_admin':50}
TEMPLATES = {
'batting_milestone': ['Rohit brings up his century with a six','Kohli reaches fifty in style',
'Dhoni scores his 10th T20I half-century'],
'bowling_wicket': ['Bumrah takes the wicket with a perfect yorker',
'Shami gets the edge and the keeper takes a simple catch'],
'fielding_dismissal':['Kohli takes a stunning catch at point','Direct hit run-out by Hardik'],
'six_scored': ['Rohit smashes a six over long-on','Massive hit clears the boundary'],
'four_scored': ['Elegant cover drive races to the boundary','Punched through covers for four'],
'dot_ball': ['Good length delivery, left alone outside off','Defended solidly back down the pitch'],
'match_admin': ['Toss won — India elect to bat','Drinks break at end of 10th over'],
}
texts_b, labels_b = [], []
for cls, n in N_PER_CLASS_B.items():
cls_idx = EVENT_CLASSES_B.index(cls)
templates = TEMPLATES[cls]
for i in range(n):
texts_b.append(templates[i % len(templates)] + f' [variant {i}]')
labels_b.append(cls_idx)
df_b = pd.DataFrame({'text': texts_b, 'label': labels_b})
df_b = df_b.sample(frac=1, random_state=42).reset_index(drop=True)
print(f'[B] Text dataset: {len(df_b)} samples')
print(f' Class distribution: {df_b["label"].value_counts().sort_index().to_dict()}')
# Class balance check (both options)
for cls_idx, count in pd.Series(y_img).value_counts().sort_index().items():
print(f' [A] {SHOT_CLASSES_A[cls_idx]}: {count} images')Step 2 — Core Logic
Step 2 builds the preprocessing pipeline and data splits. The critical rule: fit any stateful preprocessing (scalers, tokeniser vocabulary) only on the training split, never on the full dataset. For Option A, this means splitting first, then normalising each split with a scaler fitted on train. For Option B, the HuggingFace tokeniser's vocabulary is pre-trained (no fitting needed), but any custom vocabulary adaptation must follow the same rule.
Analogy🏏Cricket
🏏 Think of it like cricket: The NeuralNet class is the team management system — it tracks every player's contribution (forward pass caching), attributes match outcomes to specific decisions (backward pass gradients), and adjusts each player's role for the next match (weight update). Just as the team manager must record which batsman faced which bowler and how many runs were scored (cache activations) before attributing success or failure, the neural net must cache z and a at every layer before computing gradients. The momentum in SGD is the team's institutional memory — great past performances inform future selection policy, not overriding current evidence but weighted alongside it.
python
import numpy as np, tensorflow as tf, torch
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from transformers import AutoTokenizer
from datasets import Dataset
import pandas as pd
np.random.seed(42); tf.random.set_seed(42)
# ══════════════════════════════════════════════════════════
# OPTION A: Image Split + Normalisation
# ══════════════════════════════════════════════════════════
# Stratified split 70/15/15
X_tv, X_test_A, y_tv, y_test_A = train_test_split(
X_img, y_img, test_size=0.15, stratify=y_img, random_state=42)
X_train_A, X_val_A, y_train_A, y_val_A = train_test_split(
X_tv, y_tv, test_size=0.176, stratify=y_tv, random_state=42)
# Normalise to [0, 255] float32 — EfficientNet's preprocess_input handles /255
# No scaler needed: EfficientNet preprocess_input is stateless (divides by 255)
X_train_A = X_train_A.astype(np.float32)
X_val_A = X_val_A.astype(np.float32)
X_test_A = X_test_A.astype(np.float32)
print('[A] Split sizes:')
for name, X, y in [('Train',X_train_A,y_train_A),('Val',X_val_A,y_val_A),('Test',X_test_A,y_test_A)]:
print(f' {name}: {X.shape}, class dist: {np.bincount(y).tolist()}')
# tf.data pipeline with in-graph augmentation
BATCH_A = 32
def augment_A(image, label):
image = tf.cast(image, tf.float32) / 255.0 # [0,1]
image = tf.image.random_flip_left_right(image)
image = tf.image.random_brightness(image, 0.2)
image = tf.image.random_contrast(image, 0.8, 1.2)
image = tf.clip_by_value(image, 0, 1)
return image, label
train_ds_A = (tf.data.Dataset.from_tensor_slices((X_train_A, y_train_A))
.shuffle(1000).map(augment_A, num_parallel_calls=tf.data.AUTOTUNE)
.batch(BATCH_A).prefetch(tf.data.AUTOTUNE))
val_ds_A = (tf.data.Dataset.from_tensor_slices((X_val_A, y_val_A))
.map(lambda x,y: (tf.cast(x,tf.float32)/255.0, y))
.batch(BATCH_A).prefetch(tf.data.AUTOTUNE))
test_ds_A = (tf.data.Dataset.from_tensor_slices((X_test_A, y_test_A))
.map(lambda x,y: (tf.cast(x,tf.float32)/255.0, y))
.batch(BATCH_A).prefetch(tf.data.AUTOTUNE))
# Verify augmentation is applied at training only
for xb, yb in train_ds_A.take(1):
print(f'[A] Train batch: images {xb.shape}, labels {yb.shape}, range [{xb.numpy().min():.2f},{xb.numpy().max():.2f}]')
# ══════════════════════════════════════════════════════════
# OPTION B: Text Split + HuggingFace Dataset
# ══════════════════════════════════════════════════════════
tr_df_b, tmp_b = train_test_split(df_b, test_size=0.30, stratify=df_b['label'], random_state=42)
val_df_b, test_df_b = train_test_split(tmp_b, test_size=0.50, stratify=tmp_b['label'], random_state=42)
tokenizer_B = AutoTokenizer.from_pretrained('distilbert-base-uncased')
def tokenize_B(examples):
return tokenizer_B(examples['text'], truncation=True, max_length=64, padding='max_length')
def to_ds_B(df):
ds = Dataset.from_pandas(df.rename(columns={'label':'labels'}).reset_index(drop=True))
ds = ds.map(tokenize_B, batched=True).remove_columns(['text'])
ds.set_format('torch')
return ds
train_ds_B = to_ds_B(tr_df_b)
val_ds_B = to_ds_B(val_df_b)
test_ds_B = to_ds_B(test_df_b)
print('[B] Split sizes:')
for name, ds in [('Train',train_ds_B),('Val',val_ds_B),('Test',test_ds_B)]:
print(f' {name}: {len(ds)} samples')Step 3 — Integration & Enhancement
Step 3 verifies the complete pipeline with a sanity-check forward pass and produces the class distribution report. The sanity check confirms: data shapes are correct for the model, pixel ranges are in the expected range (0–1 for EfficientNet), labels are integers in [0, n_classes-1], and the dataloader is producing batches without error. This verification step should produce a clean output with no errors before proceeding to Lesson 33.
Analogy🏏Cricket
🏏 Think of it like cricket: Mini-batch training is like a cricket coaching session where the coach evaluates the batsman against a random set of 32 deliveries from the session's total pool, not all 500 deliveries at once. Evaluating all 500 deliveries before giving any feedback (full-batch) is slow and ignores the fact that early corrections from the first 32 balls can already improve performance on balls 33–64. Mini-batch feedback (stochastic gradients) allows the batsman to improve continuously throughout the session, converging to good technique faster than waiting for the end-of-session review. The randomness (shuffle each epoch) ensures the coach doesn't accidentally train the batsman to handle only one type of delivery sequence.
python
import numpy as np, tensorflow as tf, torch
from tensorflow import keras
from tensorflow.keras import layers
from transformers import AutoModelForSequenceClassification
np.random.seed(42)
# ══════════════════════════════════════════════════════════
# OPTION A: Pipeline verification
# ══════════════════════════════════════════════════════════
# Quick EfficientNetB0 forward pass to verify shapes
base_A = keras.applications.EfficientNetB0(
include_top=False, weights=None,
input_shape=(224,224,3), pooling='avg'
)
test_model_A = keras.Sequential([
base_A,
layers.Dense(len(SHOT_CLASSES_A), activation='softmax')
])
for xb, yb in train_ds_A.take(1):
out = test_model_A(xb, training=False)
print(f'[A] Forward pass OK: input {xb.shape} → logits {out.shape}')
print(f' Label range: [{yb.numpy().min()}, {yb.numpy().max()}] (expect 0–{len(SHOT_CLASSES_A)-1})')
print(f' Pixel range: [{xb.numpy().min():.3f}, {xb.numpy().max():.3f}] (expect 0–1)')
assert out.shape == (BATCH_A, len(SHOT_CLASSES_A)), 'Shape mismatch!'
assert yb.numpy().max() < len(SHOT_CLASSES_A), 'Label out of range!'
print('[A] Pipeline verification PASSED\n')
# ══════════════════════════════════════════════════════════
# OPTION B: Pipeline verification
# ══════════════════════════════════════════════════════════
test_model_B = AutoModelForSequenceClassification.from_pretrained(
'distilbert-base-uncased', num_labels=len(EVENT_CLASSES_B))
test_model_B.eval()
batch_b = {k: v[:4] for k, v in next(iter(
torch.utils.data.DataLoader(train_ds_B, batch_size=4))).items()}
with torch.no_grad():
out_b = test_model_B(
input_ids=batch_b['input_ids'],
attention_mask=batch_b['attention_mask']
)
print(f'[B] Forward pass OK: logits {out_b.logits.shape}')
print(f' Label range: [{batch_b["labels"].min()}, {batch_b["labels"].max()}] (expect 0–{len(EVENT_CLASSES_B)-1})')
assert out_b.logits.shape == (4, len(EVENT_CLASSES_B))
print('[B] Pipeline verification PASSED')Step 4 — Testing & Verification
Step 4 produces the data quality report that will be included in the final Capstone submission. This report must include: split sizes and class distributions for all three splits, confirmation that preprocessing was fitted on training data only, sample visualisation (3 augmented images for Option A, 3 tokenised examples with token counts for Option B), and a checklist confirming zero data leakage.
Analogy🏏Cricket
🏏 Think of it like cricket: Verifying the network is a batsman's fitness test with three non-negotiable checks. First, form must steadily improve — the loss must fall monotonically over the first 200 epochs, just as a player's error count should drop session after session, not swing wildly. Second, real skill must beat luck — test accuracy above 70% when random guessing is 50%, exactly as a batsman must clearly outscore a tail-ender swinging blindly to prove genuine technique. Third, every part of the technique must be engaged — non-zero gradient norms confirm the learning signal actually flows through all three layers, like checking footwork, backlift, and follow-through are each working rather than one part being frozen. The gold standard is the numerical gradient check: nudge a weight a tiny amount and confirm the loss changes as predicted, just as a coach films from two angles to confirm a fix is real, not imagined. The payoff: these checks prove your from-scratch network is genuinely learning and correctly wired, not just accidentally producing a decent number.
bash
import numpy as np, pandas as pd
print('=' * 60)
print('CAPSTONE PHASE 1 — DATA PIPELINE QUALITY REPORT')
print('=' * 60)
# Option A report
print('\n[OPTION A] Image Classification Pipeline')
print(f' Dataset: {X_img.shape[0]} images, {len(SHOT_CLASSES_A)} classes')
for name, y_split in [('Train',y_train_A),('Val',y_val_A),('Test',y_test_A)]:
dist = np.bincount(y_split, minlength=len(SHOT_CLASSES_A))
print(f' {name}: {len(y_split)} samples | dist: {dist.tolist()}')
print(f' Pixel range post-norm: [0.0, 1.0] ✓')
print(f' Augmentation: RandomFlip + RandomBrightness + RandomContrast ✓')
print(f' Leakage check: Normalisation is stateless (÷255) — no fitting needed ✓')
# Option B report
print('\n[OPTION B] Text Classification Pipeline')
print(f' Dataset: {len(df_b)} texts, {len(EVENT_CLASSES_B)} classes')
for name, df_split in [('Train',tr_df_b),('Val',val_df_b),('Test',test_df_b)]:
dist = df_split['label'].value_counts().sort_index()
print(f' {name}: {len(df_split)} samples | dist: {dist.tolist()}')
print(f' Tokeniser: distilbert-base-uncased (pre-trained vocab, no fitting) ✓')
print(f' Max length: 64 tokens with truncation and padding ✓')
print(f' Leakage check: Pre-trained tokeniser requires no corpus fitting ✓')
print(f' Class imbalance: dot_ball dominates — use weighted F1 and class_weight ✓')
print('\n[CHECKLIST]')
checks = [
'Dataset split is stratified (equal class proportions across splits)',
'Preprocessing fitted only on training data',
'Test split held out and NOT examined until Lesson 35',
'Forward pass sanity check passed for both options',
'Class distribution documented and imbalance strategy noted',
'Augmentation confirmed active only during training (verified in Step 3)',
]
for check in checks:
print(f' ✓ {check}')
print('\nPhase 1 COMPLETE. Proceed to Lesson 33 — Design, Train and Evaluate.')Warning: Do not examine, evaluate, or tune any model on the test split before Lesson 35's final evaluation. Store test data in a separate variable (X_test_A, test_ds_B) and add a comment marking it as 'SEALED — DO NOT USE UNTIL L35'. Any hyperparameter decision informed by test set performance — even informally — constitutes data leakage that inflates the final reported accuracy. All hyperparameter tuning in Lessons 33–34 must use only the validation split.
Extension Challenge: (1) Option A: implement online hard example mining — after each epoch, identify the 10% of training images with highest loss and sample them 3× in the next epoch. This forces the model to focus on the most confusing shot types. (2) Option B: implement curriculum learning — start training on easy examples (high zero-shot confidence) and gradually introduce hard examples (low zero-shot confidence) over epochs, mimicking how human learners progress from simple to complex tasks.
- Always split before fitting any stateful preprocessing — fit scalers, normalisation layers, and vocabulary adaptors on training data only, then transform val and test with the fitted object.
- Verify the pipeline with a sanity-check forward pass before training — catch shape mismatches, label range errors, and pixel range issues before they waste GPU training time.
- Document class distribution for all three splits — imbalanced distributions require class weighting, oversampling, or weighted metrics that must be planned before training begins.
- Seal the test split immediately after creation — no model decisions should be informed by test set performance until the final evaluation in Lesson 35.
- tf.data augmentation with num_parallel_calls=AUTOTUNE processes augmentation on CPU cores in parallel with GPU training — always use AUTOTUNE to prevent GPU starvation from data loading bottlenecks.
- Produce a written data quality report as part of Phase 1 output — this becomes Section 1 of the final Capstone project report submitted in Lesson 35.