Recurrent Neural Network
A Recurrent Neural Network (RNN) is a neural network architecture designed for sequential data, where connections loop back so that the network's hidden state at each time step depends on both the current input and the previous hidden…
Definition
A Recurrent Neural Network (RNN) is a neural network architecture designed for sequential data, where connections loop back so that the network's hidden state at each time step depends on both the current input and the previous hidden state.
Overview
Unlike feedforward networks, which process each input independently, an RNN maintains an internal hidden state that is updated at every time step as a function of the current input and the hidden state carried over from the previous step. This recurrence lets the network, in principle, retain information about earlier elements of a sequence when processing later ones, making RNNs a natural fit for tasks involving ordered data such as text, speech, and time series, where order and context matter. In practice, vanilla RNNs suffer from the vanishing and exploding gradient problem: when backpropagating error through many time steps (backpropagation through time), gradients tend to shrink toward zero or grow uncontrollably, making it very difficult for the network to learn dependencies spanning more than a handful of time steps. This limitation motivated gated architectures — Long Short-Term Memory (LSTM) networks and the simpler Gated Recurrent Unit (GRU) — which use learned gating mechanisms to control what information is retained, forgotten, or output at each step, substantially improving the ability to learn long-range dependencies. RNNs and their gated variants were the dominant architecture for sequence modeling in natural language processing and speech recognition through the mid-2010s, powering early neural machine translation and speech-to-text systems. Their inherently sequential computation, however — each step must wait for the previous one to finish — makes them slow to train on modern parallel hardware compared to architectures that process a whole sequence at once. The introduction of the Transformer architecture in 2017, with its self-attention mechanism that captures long-range dependencies without sequential recurrence and parallelizes easily across GPUs, largely displaced RNNs as the default choice for large-scale language modeling, though RNN-style architectures remain relevant in some real-time, streaming, or resource-constrained settings, and newer state-space and recurrent-inspired architectures continue to be explored as efficient alternatives to full attention.
Key Concepts
- Maintains a hidden state updated recurrently across sequential time steps
- Well suited to variable-length sequential data: text, speech, time series
- Trained via backpropagation through time (BPTT)
- Suffers from vanishing/exploding gradients over long sequences in vanilla form
- Gated variants (LSTM, GRU) mitigate long-range dependency learning problems
- Inherently sequential computation limits parallelization during training
- Largely superseded by Transformer/self-attention architectures for large-scale NLP
- Still used in streaming, real-time, or resource-constrained sequence tasks
Use Cases
Frequently Asked Questions
From the Blog
Neural Networks Explained with Simple Analogies
Neural networks are webs of simple units trained to recognise patterns — explained here without any maths.
Read More AI & TechnologyWhat Is Self-Attention in Neural Networks
Self-attention lets each token in a sequence attend to every other token in the same sequence, building context-aware representations that power transformer models.
Read More AI & TechnologyWhat Is Backpropagation in Neural Networks
Backpropagation is the algorithm that lets neural networks learn by efficiently calculating how much each weight contributed to the error and adjusting it.
Read More AI & TechnologyWhat Are Activation Functions in Neural Networks
Activation functions add non-linearity to neural networks, letting them learn complex patterns instead of behaving like a simple linear model. Here's how they work.
Read More