Activation Functions
An activation function is applied to the weighted sum of inputs at each neuron, deciding what signal (if any) is passed forward to the next layer. Without them, a neural network of any depth would reduce mathematically to a single linear transformation, because stacking linear layers just produces another linear layer. Non-linear activations are what allow networks to model curved decision boundaries, XOR-like patterns, image textures, and language semantics — essentially anything that isn't a straight line or hyperplane. The choice of activation function affects training speed, gradient flow, and the range of outputs a layer can produce.
Cricket analogy: A bowler who only ever bowls one line and length is like a network with no activation function; mixing in yorkers, bouncers, and googlies lets him carve out curved, unpredictable decision boundaries that pure straight bowling never could.
Sigmoid and Tanh
The sigmoid function squashes any real number into the range (0, 1) using 1 / (1 + e^-x), which historically made it popular for output layers in binary classification because its output can be read as a probability. Tanh is a rescaled sigmoid that outputs values in (-1, 1) and is zero-centered, which tends to help optimization slightly more than sigmoid. Both share a serious weakness: for large positive or negative inputs, their curves flatten out almost horizontally, so the gradient becomes vanishingly small. In deep networks this causes the vanishing gradient problem — early layers receive almost no learning signal because gradients shrink as they are multiplied backward through many near-zero derivatives.
Cricket analogy: Sigmoid is like a run-rate gauge squashing any scoring rate into a 0-to-1 chase probability, useful for a close finish, but once a team is cruising at 15 an over or collapsing at 2, the gauge flattens and stops giving signal, just like sigmoid's vanishing gradient at extremes.
ReLU and Its Variants
The Rectified Linear Unit, ReLU(x) = max(0, x), became the default hidden-layer activation because it is cheap to compute and does not saturate for positive inputs, so gradients flow well when a neuron is active. Its drawback is the 'dying ReLU' problem: if a neuron's weights update such that its input is always negative, it outputs zero forever and stops learning, because the gradient of ReLU is exactly zero for x < 0. Leaky ReLU fixes this by allowing a small non-zero slope (e.g. 0.01x) for negative inputs, keeping a trickle of gradient alive. Parametric ReLU (PReLU) makes that slope a learnable parameter, and ELU/GELU use smooth curves for negative inputs, often improving accuracy in modern architectures like transformers, which commonly use GELU.
Cricket analogy: ReLU is like a coach who only counts positive contributions and zeroes out negatives, cheap and effective, but a batsman in a slump gets a permanent zero rating and never recovers, mirroring dying ReLU; Leaky ReLU still credits a struggling player with a small chance lower in the order.
Softmax for Multi-Class Output
Softmax is used almost exclusively in the output layer of multi-class classifiers. It converts a vector of raw scores (logits) into a probability distribution: each output is exponentiated and divided by the sum of all exponentiated outputs, guaranteeing the results are positive and sum to 1. Because it operates on the whole output vector jointly rather than element-wise, softmax is not typically used in hidden layers — it is a final-layer readout that pairs naturally with cross-entropy loss.
Cricket analogy: Softmax is like converting raw scores of five contenders for Player of the Series into a normalized probability of winning the award, ensuring all five candidates' chances add up to exactly 100%, unlike judging each nominee independently.
import numpy as np
def sigmoid(x):
return 1 / (1 + np.exp(-x))
def relu(x):
return np.maximum(0, x)
def leaky_relu(x, alpha=0.01):
return np.where(x > 0, x, alpha * x)
def softmax(logits):
shifted = logits - np.max(logits) # numerical stability
exp = np.exp(shifted)
return exp / exp.sum()
z = np.array([-3.0, -0.5, 0.0, 2.0, 5.0])
print('sigmoid:', np.round(sigmoid(z), 3))
print('relu:', relu(z))
print('leaky_relu:', np.round(leaky_relu(z), 3))
logits = np.array([2.0, 1.0, 0.1])
print('softmax probs:', np.round(softmax(logits), 3), 'sum =', softmax(logits).sum())Think of activation functions as the 'decision gate' at each neuron. A linear gate just passes a scaled signal through — stack a thousand of them and you still only get a straight line. A non-linear gate lets each layer bend the representation, and stacking many bends is what lets deep networks carve out shapes as complex as a human face or a sentence's meaning.
A common mistake is using sigmoid or tanh in every hidden layer of a deep network 'because it worked in intro tutorials.' In practice this often causes training to stall due to vanishing gradients. Default to ReLU or GELU for hidden layers, and reserve sigmoid/softmax for the output layer depending on whether the task is binary or multi-class.
- Activation functions introduce non-linearity; without them, stacked layers collapse into one linear transform.
- Sigmoid and tanh saturate at extreme inputs, causing vanishing gradients in deep networks.
- ReLU is fast and avoids saturation for positive inputs, but can suffer from 'dying' neurons stuck at zero.
- Leaky ReLU, PReLU, ELU, and GELU are variants designed to keep gradients flowing for negative inputs.
- Softmax converts logits into a probability distribution and is used in multi-class output layers, paired with cross-entropy loss.
- Activation choice directly affects gradient flow and therefore how fast and reliably a network trains.
Practice what you learned
1. Why can't a deep neural network built entirely from linear layers (no non-linear activations) model complex, non-linear patterns?
2. What is the primary cause of the vanishing gradient problem with sigmoid and tanh activations?
3. What is the 'dying ReLU' problem?
4. Which activation function is specifically designed to output a valid probability distribution over multiple classes?
5. How does Leaky ReLU differ from standard ReLU?
Was this page helpful?
You May Also Like
What Is a Neural Network?
An introduction to neural networks as layered compositions of weighted sums and nonlinear activations, and how they learn through forward passes and training.
Training with Backpropagation
Backpropagation efficiently computes how much each weight in a neural network contributed to the error, using the chain rule to propagate gradients backward from output to input.
Gradient Descent Explained
Understand how gradient descent iteratively adjusts model parameters to minimize a loss function, and how learning rate and variants like SGD affect convergence.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.