Neural Networks Basics Cheat Sheet
A reference for foundational neural network concepts covering feedforward architectures in PyTorch and Keras, backpropagation, activations, and regularization.
2 PagesBeginnerMar 10, 2026
Feedforward Network in PyTorch
Define a simple multilayer perceptron.
python
import torchimport torch.nn as nnclass MLP(nn.Module): def __init__(self, in_dim, hidden_dim, out_dim): super().__init__() self.net = nn.Sequential( nn.Linear(in_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, out_dim) ) def forward(self, x): return self.net(x)model = MLP(784, 128, 10)
Training Loop
The standard PyTorch train step pattern.
python
criterion = nn.CrossEntropyLoss()optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)for epoch in range(10): for X_batch, y_batch in train_loader: optimizer.zero_grad() outputs = model(X_batch) loss = criterion(outputs, y_batch) loss.backward() # backpropagation optimizer.step()
Keras Equivalent
The same network defined with the Keras API.
python
from tensorflow import kerasmodel = keras.Sequential([ keras.layers.Dense(128, activation='relu', input_shape=(784,)), keras.layers.Dense(10, activation='softmax')])model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])model.fit(X_train, y_train, epochs=10, batch_size=32, validation_split=0.1)
Key Concepts
Core theory behind neural networks.
- Activation function- Introduces non-linearity (ReLU, sigmoid, tanh); without it, stacked layers collapse into one linear function
- Backpropagation- Computes the loss gradient with respect to every weight via the chain rule, layer by layer
- Weight initialization- Poor initialization (e.g. all zeros) causes symmetric, non-learning neurons; use He or Xavier initialization
- Learning rate- Step size for gradient updates; too high diverges, too low converges very slowly
- Overfitting- Combat with dropout, weight decay (L2), early stopping, or more training data
- Batch size- Number of samples per gradient update; affects training stability, speed, and memory use
Backpropagation by Hand (NumPy)
Implementing forward and backward passes manually clarifies what autograd does under the hood.
python
import numpy as npdef sigmoid(z): return 1 / (1 + np.exp(-z))def sigmoid_deriv(a): return a * (1 - a) # derivative in terms of the activation# Forward passz1 = X @ W1 + b1a1 = sigmoid(z1)z2 = a1 @ W2 + b2a2 = sigmoid(z2) # predictionloss = np.mean((a2 - y) ** 2)# Backward pass (chain rule, layer by layer)dL_da2 = 2 * (a2 - y) / y.shape[0]dL_dz2 = dL_da2 * sigmoid_deriv(a2)dL_dW2 = a1.T @ dL_dz2dL_da1 = dL_dz2 @ W2.TdL_dz1 = dL_da1 * sigmoid_deriv(a1)dL_dW1 = X.T @ dL_dz1W1 -= lr * dL_dW1; W2 -= lr * dL_dW2
Custom Autograd Function in PyTorch
Extend autograd with a hand-written forward/backward when you need a non-standard op.
python
import torchclass HardSigmoid(torch.autograd.Function): @staticmethod def forward(ctx, x): ctx.save_for_backward(x) return torch.clamp((x + 3) / 6, 0, 1) @staticmethod def backward(ctx, grad_output): (x,) = ctx.saved_tensors grad_input = grad_output.clone() grad_input[(x < -3) | (x > 3)] = 0 # zero gradient outside the linear region return grad_inputx = torch.randn(4, requires_grad=True)out = HardSigmoid.apply(x)out.sum().backward()
LR Schedulers, Warmup, and Gradient Clipping
Production training loops rarely use a fixed learning rate or unclipped gradients.
python
import torchoptimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)scheduler = torch.optim.lr_scheduler.OneCycleLR( optimizer, max_lr=3e-4, total_steps=len(train_loader) * epochs)for epoch in range(epochs): for X_batch, y_batch in train_loader: optimizer.zero_grad() loss = criterion(model(X_batch), y_batch) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # avoid exploding grads optimizer.step() scheduler.step()
Explicit Weight Initialization
Override PyTorch's default initialization to match the activation function you're using.
python
import torch.nn as nndef init_weights(module): if isinstance(module, nn.Linear): # He/Kaiming init pairs with ReLU-family activations nn.init.kaiming_normal_(module.weight, nonlinearity='relu') nn.init.zeros_(module.bias)model = MLP(784, 128, 10)model.apply(init_weights)# Xavier/Glorot init is the better fit for tanh/sigmoid activations:# nn.init.xavier_uniform_(module.weight)
Advanced Training Concepts
Beyond the basic forward/backward/update loop.
- Vanishing/exploding gradients- Gradients shrink or blow up multiplicatively across many layers; mitigate with normalization, residual connections, and careful initialization
- Batch normalization- Normalizes layer inputs per mini-batch, stabilizing and accelerating training; behaves differently at train vs. eval time (running stats)
- Layer normalization- Normalizes across features instead of the batch dimension; batch-size independent, standard in transformers
- Learning rate warmup- Linearly ramp the LR up from ~0 for the first few hundred steps to avoid destabilizing randomly-initialized weights
- Mixed precision training- Uses float16/bfloat16 for most ops with a float32 master copy of weights, roughly halving memory and increasing throughput on modern GPUs
- Gradient accumulation- Sums gradients over several small forward/backward passes before an optimizer step, simulating a larger batch size under memory constraints
- Dead ReLU- Neurons stuck outputting 0 for all inputs because their weights drove them into ReLU's zero-gradient region; LeakyReLU/GELU avoid this
Pro Tip
Vanishing gradients in deep networks are often a symptom of saturating sigmoid or tanh activations — switch to ReLU (or variants like LeakyReLU or GELU) and add batch normalization to keep gradients flowing through deep stacks.
Was this cheat sheet helpful?
Explore Topics
#NeuralNetworksBasics#NeuralNetworksBasicsCheatSheet#DataScience#Beginner#FeedforwardNetworkInPyTorch#TrainingLoop#KerasEquivalent#KeyConcepts#Networking#MachineLearning#CheatSheet#SkillVeris
