Recurrent Neural Networks Cheat Sheet
A reference for Recurrent Neural Networks covering LSTM and GRU implementations, sequence padding, vanishing gradients, and bidirectional architectures.
2 PagesIntermediateMar 5, 2026
LSTM in PyTorch
A sequence classifier built on an embedding and LSTM.
python
import torch.nn as nnclass LSTMClassifier(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes): super().__init__() self.embedding = nn.Embedding(vocab_size, embed_dim) self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True, num_layers=2) self.fc = nn.Linear(hidden_dim, num_classes) def forward(self, x): embedded = self.embedding(x) output, (hidden, cell) = self.lstm(embedded) return self.fc(hidden[-1]) # last layer's final hidden state
GRU in Keras
A lighter-weight recurrent layer for sequence classification.
python
from tensorflow.keras import layers, modelsmodel = models.Sequential([ layers.Embedding(input_dim=10000, output_dim=64), layers.GRU(64, return_sequences=False), layers.Dense(1, activation='sigmoid')])model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
Sequence Padding
Normalize variable-length sequences to a fixed length.
python
from tensorflow.keras.preprocessing.sequence import pad_sequencespadded = pad_sequences(sequences, maxlen=100, padding='post', truncating='post')
Key Concepts
Core theory behind RNNs.
- Hidden state- Carries information across time steps, updated at each step from the input and previous state
- Vanishing/exploding gradients- Backpropagation through time over many steps can shrink or blow up gradients in vanilla RNNs
- LSTM- Uses input, forget, and output gates plus a separate cell state to preserve long-range dependencies
- GRU- Simplified gating (update and reset gates) with fewer parameters than LSTM, often comparable performance
- Bidirectional RNN- Processes the sequence forward and backward, useful whenever full context is available upfront
- Teacher forcing- Feeds the true previous token during training instead of the model's own prediction, stabilizing learning
Custom RNN Cell From Scratch
Implementing the raw recurrence relation shows exactly what nn.LSTM abstracts away.
python
import torchimport torch.nn as nnclass VanillaRNNCell(nn.Module): def __init__(self, input_dim, hidden_dim): super().__init__() self.W_ih = nn.Linear(input_dim, hidden_dim) self.W_hh = nn.Linear(hidden_dim, hidden_dim, bias=False) def forward(self, x_t, h_prev): # h_t = tanh(W_ih x_t + W_hh h_{t-1}) return torch.tanh(self.W_ih(x_t) + self.W_hh(h_prev))cell = VanillaRNNCell(input_dim=32, hidden_dim=64)h = torch.zeros(1, 64)for x_t in torch.unbind(inputs, dim=1): # inputs: (batch, seq_len, input_dim) h = cell(x_t, h)# h now holds the final hidden state after manually unrolling time steps
Gradient Clipping for BPTT
Clip gradient norms before the optimizer step to keep truncated backprop-through-time stable.
python
import torch.nn.utils as utilsoptimizer.zero_grad()loss = criterion(model(x_batch), y_batch)loss.backward()# Clip the global norm across all parameters, not per-tensorutils.clip_grad_norm_(model.parameters(), max_norm=1.0)optimizer.step()# For very long sequences, truncate BPTT by detaching the hidden state# every k steps so gradients don't flow past the truncation window:hidden = tuple(h.detach() for h in hidden)
PackedSequence for Variable-Length Batches
Avoid wasting compute on padding tokens by packing before feeding the LSTM.
python
import torch.nn as nnfrom torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence# lengths: true (unpadded) length of each sequence in the batch, sorted descendingpacked = pack_padded_sequence(embedded, lengths, batch_first=True, enforce_sorted=False)packed_out, (h_n, c_n) = lstm(packed)output, out_lengths = pad_packed_sequence(packed_out, batch_first=True)# output is repadded to (batch, max_len, hidden_dim); PyTorch skips compute# on padded positions internally, which matters a lot for long, ragged batches
Seq2Seq Decoder With Bahdanau Attention
The attention mechanism that predates Transformers, still used in small encoder-decoder RNNs.
python
import torchimport torch.nn as nnimport torch.nn.functional as Fclass BahdanauAttention(nn.Module): def __init__(self, hidden_dim): super().__init__() self.W_enc = nn.Linear(hidden_dim, hidden_dim, bias=False) self.W_dec = nn.Linear(hidden_dim, hidden_dim, bias=False) self.v = nn.Linear(hidden_dim, 1, bias=False) def forward(self, decoder_h, encoder_outputs): # decoder_h: (batch, hidden), encoder_outputs: (batch, src_len, hidden) query = self.W_dec(decoder_h).unsqueeze(1) # (batch, 1, hidden) keys = self.W_enc(encoder_outputs) # (batch, src_len, hidden) scores = self.v(torch.tanh(query + keys)).squeeze(-1) # (batch, src_len) weights = F.softmax(scores, dim=-1) context = torch.bmm(weights.unsqueeze(1), encoder_outputs).squeeze(1) return context, weights
Advanced Gotchas
Failure modes that surface once you move past toy sequence-classification examples.
- Exploding gradients vs. vanishing- exploding gradients spike loss to NaN and are fixed with clip_grad_norm_; vanishing gradients silently stop learning on long-range dependencies and need gated units or shorter BPTT windows
- Stateful vs. stateless RNNs- a stateful RNN carries hidden state across batches (must not shuffle data, must call reset_states() at epoch boundaries in Keras); stateless resets every batch
- Peephole connections- variant LSTM where gates can also see the cell state directly, occasionally improves precise timing tasks but is rarely available in high-level APIs
- Layer normalization in RNNs- batch norm doesn't work well across time steps of variable length; LayerNorm (applied per time step, per sample) is the standard normalization choice inside recurrent cells
- Cudnn fused kernels- nn.LSTM/nn.GRU on CUDA use fused cuDNN kernels that are much faster than manually unrolled loops, but they require consistent dtypes/contiguous memory and disable custom per-step logic
- Attention-augmented RNNs- adding an attention layer over encoder outputs (as in Bahdanau/Luong attention) largely fixed the fixed-context-vector bottleneck before Transformers replaced recurrence entirely
Pro Tip
Reach for LSTMs or GRUs mainly when compute or context length is tightly constrained — Transformer-based architectures have largely superseded vanilla RNNs for tasks with longer-range dependencies because self-attention avoids the sequential bottleneck and vanishing-gradient issues RNNs face.
Was this cheat sheet helpful?
Explore Topics
#RecurrentNeuralNetworks#RecurrentNeuralNetworksCheatSheet#DataScience#Intermediate#LSTMInPyTorch#GRUInKeras#SequencePadding#KeyConcepts#MachineLearning#CheatSheet#SkillVeris
