Variational Autoencoder
A Variational Autoencoder (VAE) is a generative neural network that learns a probabilistic latent representation of data, enabling it to both reconstruct inputs and generate new, plausible samples by sampling from a structured latent space.
Definition
A Variational Autoencoder (VAE) is a generative neural network that learns a probabilistic latent representation of data, enabling it to both reconstruct inputs and generate new, plausible samples by sampling from a structured latent space.
Overview
A VAE extends the basic autoencoder architecture by making its latent space probabilistic rather than a fixed point. Instead of encoding an input directly into a single latent vector, the encoder outputs the parameters (mean and variance) of a probability distribution, typically Gaussian, and a latent vector is then sampled from that distribution before being passed to the decoder. To make this sampling step compatible with gradient-based training, VAEs use the "reparameterization trick," which separates the randomness from the network parameters so gradients can flow through the sampling operation during backpropagation. VAEs are trained with a loss function combining two terms: a reconstruction loss, similar to a standard autoencoder, that measures how well the decoder reconstructs the original input, and a Kullback-Leibler (KL) divergence term that regularizes the learned latent distribution to stay close to a simple prior, usually a standard normal distribution. This regularization is what gives VAEs their key generative property: because the latent space is encouraged to be smooth and continuous rather than scattered arbitrarily, sampling a random point from the prior distribution and passing it through the decoder reliably produces a plausible, novel output, unlike a standard autoencoder's latent space, which has no such guarantee. Introduced in a 2013 paper by Kingma and Welling, VAEs were among the first practical deep generative models capable of both fast, direct generation and meaningful, structured latent representations, making them useful for tasks requiring latent space interpolation and manipulation, such as smoothly morphing between two generated faces. Compared to Generative Adversarial Networks (GANs), VAEs tend to produce blurrier outputs, particularly for images, because the reconstruction loss encourages averaging over plausible outputs, but they offer more stable training and an explicit, interpretable probabilistic framework. VAEs remain influential in areas like drug discovery, anomaly detection, and as components within larger generative systems, including some diffusion model architectures that use a VAE to compress images into a latent space before applying the diffusion process.
Key Concepts
- Encodes input into a probability distribution over the latent space, not a fixed point
- Uses the reparameterization trick to enable gradient-based training through sampling
- Loss combines reconstruction error with a KL divergence regularization term
- Latent space is smooth and continuous, enabling meaningful interpolation
- Capable of generating novel samples by sampling from the prior distribution
- Introduced by Kingma and Welling in 2013
- Generally produces blurrier outputs than GANs but trains more stably
- Used as a compression component within some diffusion model architectures