Practice — build a neural net from scratch in NumPy

What You'll Build

You will build a fully functional 3-layer neural network from scratch using only NumPy — no PyTorch, no TensorFlow, no shortcuts. The network will predict whether an Indian Premier League (IPL) team will win a match, given batting and bowling statistics for both teams. You will implement the forward pass (matrix multiplications, ReLU, sigmoid), the backward pass (chain rule, gradient computation for every weight and bias), the weight update step (SGD with momentum), and a training loop that monitors loss and accuracy. By the end you will have a working neural net you built yourself — giving you ground-level understanding of exactly what happens inside every framework. This is the foundation that makes debugging, optimising, and architecting real networks intuitive rather than magical.

Analogy🏏Cricket

🏏 Think of it like cricket: Building a neural net from scratch is like a young cricketer learning the game by playing gully cricket with a tape-ball before ever stepping into a coaching academy. You don't use a bowling machine, you don't get video analysis, you don't have a structured training manual — you just play, make mistakes, adapt. Just as gully cricket teaches the core instincts of timing, footwork, and reading the ball that no academy drill can fully replicate, building a network from NumPy teaches the core mechanics of gradient flow, matrix shapes, and numerical stability that no framework hides. Just as every Indian international cricketer traces their instincts back to gully cricket roots, every deep learning practitioner benefits from having once implemented backpropagation themselves. The insight is that frameworks automate what you understand — building from scratch ensures you actually understand it.

Prerequisites

NumPy basics: array creation, matrix multiplication (@), broadcasting, element-wise operations
Understanding of forward pass: activation(W @ x + b) for each layer
Understanding of backpropagation: chain rule, dL/dW = activation_upstream × local_input
Familiarity with sigmoid and ReLU activation functions and their derivatives
Basic Python: classes, for loops, list comprehensions

Setup & Project Structure

This exercise requires only NumPy — no additional libraries. Create a single Python file cricket_neural_net.py. The project will have four natural sections: data generation (synthetic IPL match statistics), the NeuralNet class (forward + backward + update), the training loop (loss monitoring, accuracy tracking), and evaluation (test set performance). Keep all code in one file for clarity — the goal is to see the entire system at once, not to engineer a production codebase.

Analogy🏏Cricket

🏏 Think of it like cricket: Before a net session you lay out the ground in clear zones — the bowling machine and pitch (data generation of synthetic IPL stats), the batsman's technique broken into stance, backlift, and follow-through (the NeuralNet forward, backward, and update methods), the throwdown routine that repeats and tracks improvement (the training loop with loss and accuracy monitoring), and finally a match simulation to test readiness (test-set evaluation). Just as a coach keeps the whole drill on one ground so a player can see how each phase connects, keeping all the code in one file lets you trace how a forward pass flows into a gradient and then a weight update. Just as you need only bat, ball, and pitch — not a full stadium — for a productive net, this exercise needs only NumPy, no heavy libraries. The payoff: a clean, well-sectioned practice structure means every mechanic is visible and debuggable, so you understand exactly why the network learns rather than treating it as a black box.

bash

# Setup — no pip installs needed, only NumPy
import numpy as np

# Verify NumPy version
print(f'NumPy version: {np.__version__}')

# Project structure (single file):
# cricket_neural_net.py
# ├── generate_ipl_data()       — synthetic match stats
# ├── class NeuralNet           — forward, backward, update
# │   ├── __init__()            — weight initialisation
# │   ├── forward()             — forward pass, cache activations
# │   ├── backward()            — backprop, compute all gradients
# │   └── update()              — SGD+momentum weight update
# ├── train()                   — training loop with loss logging
# └── evaluate()                — accuracy on test set

print('Structure ready — begin implementation')

Step 1 — Foundation

Step 1 builds the data layer and weight initialisation. We generate synthetic IPL match data: 6 features per match (home batting average, home bowling economy, home run rate, away batting average, away bowling economy, away run rate) and a binary label (1 = home team wins). We use He initialisation for the two ReLU hidden layers and Xavier for the sigmoid output layer. Correct initialisation is the first prerequisite for training convergence — initialise wrong and no amount of tuning will save the model.

Analogy🏏Cricket

🏏 Think of it like cricket: Before the first ball is bowled, the groundsman prepares the pitch — the pitch condition is the foundation that determines what is possible. Bad pitch preparation (wrong initialisation) makes even the best bowlers and batsmen ineffective. Just as a pitch that is too green (over-seamed) or too dry (over-spun) constrains the entire match, bad weight initialisation constrains the entire training run. He initialisation is the 'neutral pitch' — well-prepared, giving both batting and bowling a fair contest, from which any outcome is possible.

python

import numpy as np
np.random.seed(42)

def generate_ipl_data(n_matches=1000):
    """
    Synthetic IPL match statistics.
    Features: [home_batting_avg, home_economy, home_run_rate,
               away_batting_avg, away_economy, away_run_rate]
    Label: 1 = home team wins
    """
    # Home team stats — slightly biased toward higher batting avg (home advantage)
    home_batting_avg = np.random.normal(32, 8, n_matches)
    home_economy     = np.random.normal(8.2, 1.5, n_matches)
    home_run_rate    = np.random.normal(8.5, 1.2, n_matches)

    # Away team stats
    away_batting_avg = np.random.normal(30, 8, n_matches)
    away_economy     = np.random.normal(8.5, 1.5, n_matches)
    away_run_rate    = np.random.normal(8.0, 1.2, n_matches)

    X = np.column_stack([
        home_batting_avg, home_economy, home_run_rate,
        away_batting_avg, away_economy, away_run_rate
    ])  # shape (N, 6)

    # Label: home wins if batting avg gap + run rate gap > 0
    batting_edge  = home_batting_avg - away_batting_avg
    economy_edge  = away_economy - home_economy        # lower economy = better
    run_rate_edge = home_run_rate - away_run_rate
    raw_score = batting_edge * 0.5 + economy_edge * 0.3 + run_rate_edge * 0.4
    y = (raw_score + np.random.randn(n_matches) * 3 > 0).astype(float)  # shape (N,)

    # Normalise features to mean=0, std=1 for stable training
    X = (X - X.mean(axis=0)) / (X.std(axis=0) + 1e-8)
    return X, y

def he_init(fan_in, fan_out):
    return np.random.randn(fan_out, fan_in) * np.sqrt(2.0 / fan_in)

def xavier_init(fan_in, fan_out):
    return np.random.randn(fan_out, fan_in) * np.sqrt(2.0 / (fan_in + fan_out))

X, y = generate_ipl_data(1000)
print(f'Dataset shape: X={X.shape}, y={y.shape}')
print(f'Win rate: {y.mean():.2%}')  # should be ~50% (balanced)

# Weight initialisation
input_dim, hidden1, hidden2, output_dim = 6, 16, 8, 1
W1 = he_init(input_dim, hidden1);    b1 = np.zeros(hidden1)
W2 = he_init(hidden1,  hidden2);    b2 = np.zeros(hidden2)
W3 = xavier_init(hidden2, output_dim); b3 = np.zeros(output_dim)
print(f'W1:{W1.shape} W2:{W2.shape} W3:{W3.shape}')

Step 2 — Core Logic

Step 2 implements the complete NeuralNet class with forward pass, backward pass, and weight update. The forward pass must cache all intermediate values for backpropagation. The backward pass implements the chain rule layer by layer, computing dL/dW and dL/db for every parameter. The update step applies SGD with momentum. The full class is the heart of the exercise — every line corresponds directly to a mathematical operation you covered in the theory lessons.

Analogy🏏Cricket

🏏 Think of it like cricket: The NeuralNet class is the team management system — it tracks every player's contribution (forward pass caching), attributes match outcomes to specific decisions (backward pass gradients), and adjusts each player's role for the next match (weight update). Just as the team manager must record which batsman faced which bowler and how many runs were scored (cache activations) before attributing success or failure, the neural net must cache z and a at every layer before computing gradients. The momentum in SGD is the team's institutional memory — great past performances inform future selection policy, not overriding current evidence but weighted alongside it.

python

import numpy as np
np.random.seed(42)

class IPLWinPredictor:
    """3-layer neural net from scratch — no frameworks."""

    def __init__(self, input_dim=6, h1=16, h2=8, lr=0.05, momentum=0.9):
        self.lr  = lr
        self.mom = momentum

        # He init for ReLU layers, Xavier for sigmoid output
        self.W1 = np.random.randn(h1, input_dim) * np.sqrt(2 / input_dim)
        self.b1 = np.zeros(h1)
        self.W2 = np.random.randn(h2, h1)        * np.sqrt(2 / h1)
        self.b2 = np.zeros(h2)
        self.W3 = np.random.randn(1,  h2)        * np.sqrt(2 / (h2 + 1))
        self.b3 = np.zeros(1)

        # Momentum velocity terms
        self.vW1 = np.zeros_like(self.W1)
        self.vW2 = np.zeros_like(self.W2)
        self.vW3 = np.zeros_like(self.W3)
        self.vb1 = np.zeros_like(self.b1)
        self.vb2 = np.zeros_like(self.b2)
        self.vb3 = np.zeros_like(self.b3)

    # ── Activations ──────────────────────────────────────────
    @staticmethod
    def relu(z):      return np.maximum(0, z)
    @staticmethod
    def relu_grad(z): return (z > 0).astype(float)
    @staticmethod
    def sigmoid(z):   return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

    # ── Forward pass — MUST cache all intermediate values ────
    def forward(self, X):
        self.X  = X
        self.z1 = X    @ self.W1.T + self.b1;  self.a1 = self.relu(self.z1)
        self.z2 = self.a1 @ self.W2.T + self.b2;  self.a2 = self.relu(self.z2)
        self.z3 = self.a2 @ self.W3.T + self.b3;  self.a3 = self.sigmoid(self.z3)
        return self.a3  # shape (N, 1)

    # ── Backward pass — chain rule through all layers ────────
    def backward(self, y_true):
        N = self.X.shape[0]
        y = y_true.reshape(-1, 1)

        # Output layer: d(BCE)/d(z3) = a3 - y  [simplified]
        dz3 = self.a3 - y                          # (N, 1)
        self.dW3 = dz3.T @ self.a2 / N            # (1, 8)
        self.db3 = dz3.mean(axis=0)               # (1,)

        # Hidden layer 2
        da2      = dz3 @ self.W3                  # (N, 8)
        dz2      = da2 * self.relu_grad(self.z2)  # (N, 8)
        self.dW2 = dz2.T @ self.a1 / N           # (8, 16)
        self.db2 = dz2.mean(axis=0)              # (8,)

        # Hidden layer 1
        da1      = dz2 @ self.W2                  # (N, 16)
        dz1      = da1 * self.relu_grad(self.z1)  # (N, 16)
        self.dW1 = dz1.T @ self.X  / N           # (16, 6)
        self.db1 = dz1.mean(axis=0)              # (16,)

    # ── SGD + momentum update ────────────────────────────────
    def update(self):
        for W, dW, vW, b, db, vb in [
            (self.W1,self.dW1,self.vW1, self.b1,self.db1,self.vb1),
            (self.W2,self.dW2,self.vW2, self.b2,self.db2,self.vb2),
            (self.W3,self.dW3,self.vW3, self.b3,self.db3,self.vb3),
        ]:
            vW[:] = self.mom * vW + self.dW
            W    -= self.lr  * vW
            vb[:] = self.mom * vb + self.db
            b    -= self.lr  * vb

    def loss(self, y_true):
        eps = 1e-8
        p = np.clip(self.a3, eps, 1-eps).flatten()
        return -np.mean(y_true * np.log(p) + (1-y_true) * np.log(1-p))

    def predict(self, X):
        return (self.forward(X).flatten() > 0.5).astype(float)

print('IPLWinPredictor class defined — proceed to training')

Step 3 — Integration & Enhancement

Step 3 brings everything together: generate the dataset, split into train/test, run the training loop for 500 epochs with mini-batches, track loss and accuracy, and evaluate on the held-out test set. Mini-batch training is more realistic than full-batch gradient descent — it introduces the beneficial noise of stochastic updates and is how all real networks are trained. We use a batch size of 32, consistent with common production defaults.

Analogy🏏Cricket

🏏 Think of it like cricket: Mini-batch training is like a cricket coaching session where the coach evaluates the batsman against a random set of 32 deliveries from the session's total pool, not all 500 deliveries at once. Evaluating all 500 deliveries before giving any feedback (full-batch) is slow and ignores the fact that early corrections from the first 32 balls can already improve performance on balls 33–64. Mini-batch feedback (stochastic gradients) allows the batsman to improve continuously throughout the session, converging to good technique faster than waiting for the end-of-session review. The randomness (shuffle each epoch) ensures the coach doesn't accidentally train the batsman to handle only one type of delivery sequence.

python

import numpy as np
np.random.seed(42)

# Paste generate_ipl_data() and IPLWinPredictor class from Steps 1 & 2 here
# Then run the full training loop below:

def generate_ipl_data(n_matches=1000):
    home_batting_avg = np.random.normal(32, 8, n_matches)
    home_economy     = np.random.normal(8.2, 1.5, n_matches)
    home_run_rate    = np.random.normal(8.5, 1.2, n_matches)
    away_batting_avg = np.random.normal(30, 8, n_matches)
    away_economy     = np.random.normal(8.5, 1.5, n_matches)
    away_run_rate    = np.random.normal(8.0, 1.2, n_matches)
    X = np.column_stack([home_batting_avg, home_economy, home_run_rate,
                         away_batting_avg, away_economy, away_run_rate])
    raw_score = (home_batting_avg - away_batting_avg)*0.5 + \
                (away_economy - home_economy)*0.3 + \
                (home_run_rate - away_run_rate)*0.4
    y = (raw_score + np.random.randn(n_matches)*3 > 0).astype(float)
    X = (X - X.mean(axis=0)) / (X.std(axis=0) + 1e-8)
    return X, y

X, y = generate_ipl_data(1000)
# Train / test split: 80% / 20%
split = int(0.8 * len(X))
idx   = np.random.permutation(len(X))
X_train, y_train = X[idx[:split]], y[idx[:split]]
X_test,  y_test  = X[idx[split:]], y[idx[split:]]

model = IPLWinPredictor(lr=0.05, momentum=0.9)
batch_size = 32
epochs     = 500

for innings_count in range(epochs):
    # Shuffle training data each epoch
    perm = np.random.permutation(len(X_train))
    X_train_s, y_train_s = X_train[perm], y_train[perm]

    # Mini-batch gradient descent
    for i in range(0, len(X_train_s), batch_size):
        Xb = X_train_s[i:i+batch_size]
        yb = y_train_s[i:i+batch_size]
        model.forward(Xb)
        model.backward(yb)
        model.update()

    # Monitor every 100 epochs
    if innings_count % 100 == 0:
        model.forward(X_train)
        train_loss = model.loss(y_train)
        train_acc  = (model.predict(X_train) == y_train).mean()
        val_acc    = (model.predict(X_test)  == y_test).mean()
        print(f'Epoch {innings_count:4d} | Loss: {train_loss:.4f} | '
              f'Train Acc: {train_acc:.2%} | Val Acc: {val_acc:.2%}')

# Final evaluation
final_val_acc = (model.predict(X_test) == y_test).mean()
print(f'\nFinal test accuracy: {final_val_acc:.2%}')

Step 4 — Testing & Verification

Verify the implementation is correct by running the script and checking three things: loss must decrease monotonically over the first 200 epochs, final test accuracy must exceed 70% (random would be 50%), and gradient norms must not be zero (confirming gradients actually flow through all three layers). A numerical gradient check is the gold-standard verification — compute finite-difference approximations and compare to analytical gradients.

Analogy🏏Cricket

🏏 Think of it like cricket: Verifying the network is a batsman's fitness test with three non-negotiable checks. First, form must steadily improve — the loss must fall monotonically over the first 200 epochs, just as a player's error count should drop session after session, not swing wildly. Second, real skill must beat luck — test accuracy above 70% when random guessing is 50%, exactly as a batsman must clearly outscore a tail-ender swinging blindly to prove genuine technique. Third, every part of the technique must be engaged — non-zero gradient norms confirm the learning signal actually flows through all three layers, like checking footwork, backlift, and follow-through are each working rather than one part being frozen. The gold standard is the numerical gradient check: nudge a weight a tiny amount and confirm the loss changes as predicted, just as a coach films from two angles to confirm a fix is real, not imagined. The payoff: these checks prove your from-scratch network is genuinely learning and correctly wired, not just accidentally producing a decent number.

bash

# Run the full training script and check output:
# Expected output:
# Epoch    0 | Loss: 0.69xx | Train Acc: ~50% | Val Acc: ~50%
# Epoch  100 | Loss: 0.6xxx | Train Acc: ~60% | Val Acc: ~58%
# Epoch  200 | Loss: 0.5xxx | Train Acc: ~67% | Val Acc: ~65%
# Epoch  300 | Loss: 0.4xxx | Train Acc: ~72% | Val Acc: ~70%
# Epoch  400 | Loss: 0.4xxx | Train Acc: ~75% | Val Acc: ~72%
# Final test accuracy: ~72-76%

# Gradient check — verify backprop is correct
def numerical_gradient(model, X, y, eps=1e-5):
    """Finite-difference gradient for W3 — compare to analytical."""
    model.forward(X); model.backward(y)
    analytic_dW3 = model.dW3.copy()

    numerical_dW3 = np.zeros_like(model.W3)
    for i in range(model.W3.shape[0]):
        for j in range(model.W3.shape[1]):
            model.W3[i,j] += eps
            model.forward(X); loss_plus = model.loss(y)
            model.W3[i,j] -= 2*eps
            model.forward(X); loss_minus = model.loss(y)
            model.W3[i,j] += eps  # restore
            numerical_dW3[i,j] = (loss_plus - loss_minus) / (2*eps)

    err = np.max(np.abs(analytic_dW3 - numerical_dW3))
    print(f'Gradient check max error: {err:.2e}  (should be < 1e-5)')
    return err

# Use a small sample for speed
err = numerical_gradient(model, X_train[:10], y_train[:10])
assert err < 1e-4, f'Backprop may have a bug! Error: {err}'

Warning: If your final test accuracy is stuck at exactly 50% after 500 epochs, check three things in order: (1) Is your loss function correct? Log(0) returning -inf will NaN your gradients immediately. (2) Did you forget to divide gradients by N (batch size) in the backward pass? Without this, gradient magnitudes scale with batch size and cause oscillation. (3) Did you shuffle training data each epoch? Without shuffling, mini-batches are always the same and the model memorises batch order rather than learning features.

Extension Challenge: (1) Add L2 regularisation to the loss function (lambda * sum(W²)) and to the gradient (dW += lambda * W) and observe the effect on overfitting. (2) Implement Adam from scratch inside the IPLWinPredictor class and compare convergence speed to SGD+momentum. (3) Add a fourth hidden layer and observe whether deeper improves or hurts accuracy on this dataset — explain why using what you know about vanishing gradients.

Always normalise input features to mean=0, std=1 before training — unnormalised features cause gradients of very different magnitudes that destabilise training.
Cache all intermediate values (X, z1, a1, z2, a2, z3, a3) during the forward pass — backpropagation needs them to compute weight gradients via the chain rule.
The output gradient dz3 = a3 - y is the simplified chain rule derivative of BCE + sigmoid — always -1 to 1, perfectly conditioned for gradient descent.
Divide all weight gradients by N (batch size) to compute the mean gradient — without this, gradient magnitude scales with batch size causing inconsistent step sizes.
Use He initialisation for ReLU layers (std=sqrt(2/fan_in)) and Xavier for sigmoid/tanh layers — wrong initialisation is the most common cause of training that never starts.
Mini-batch training (batch_size=32) with shuffling each epoch introduces beneficial gradient noise that helps escape local minima compared to full-batch gradient descent.