Generative Adversarial Networks (GANs) Cheat Sheet
Explains the generator-discriminator minimax game, common failure modes like mode collapse, and a minimal PyTorch training loop for a GAN.
2 PagesAdvancedMar 5, 2026
Core Concepts
The adversarial training setup.
- Generator (G)- Maps random noise z from a latent space to a synthetic sample meant to look real
- Discriminator (D)- Binary classifier trained to distinguish real samples from G's fake samples
- Minimax objective- min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]; G and D are trained with opposing goals
- Latent space- The input noise distribution (often standard normal) that G transforms into data
- Mode collapse- G learns to produce only a few varieties of output that reliably fool D, losing diversity
- Nash equilibrium- The theoretical training target where D can no longer distinguish real from fake (D outputs 0.5 everywhere)
Minimal GAN Training Loop
Alternating updates to the discriminator and generator.
python
import torch, torch.nn as nncriterion = nn.BCELoss()opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))for real_batch in dataloader: batch_size = real_batch.size(0) real_labels = torch.ones(batch_size, 1) fake_labels = torch.zeros(batch_size, 1) # --- Train Discriminator --- z = torch.randn(batch_size, latent_dim) fake_batch = G(z) d_loss = criterion(D(real_batch), real_labels) + \ criterion(D(fake_batch.detach()), fake_labels) opt_d.zero_grad(); d_loss.backward(); opt_d.step() # --- Train Generator --- g_loss = criterion(D(fake_batch), real_labels) # wants D to say "real" opt_g.zero_grad(); g_loss.backward(); opt_g.step()
Common GAN Variants
Architectures that address specific weaknesses of the vanilla GAN.
- DCGAN- Uses convolutional/transposed-convolutional layers with batch norm for stable image generation
- WGAN- Replaces the JS-divergence-based loss with the Wasserstein distance and weight clipping/gradient penalty for more stable training
- Conditional GAN (cGAN)- Conditions both G and D on a label or class so generation can be controlled
- CycleGAN- Learns unpaired image-to-image translation using cycle-consistency loss (no paired training data required)
- StyleGAN- Injects latent style vectors at multiple resolutions for fine-grained control over generated image attributes
Stabilizing GAN Training
GANs are notoriously unstable to train.
- Label smoothing- Use 0.9 instead of 1.0 for real labels to prevent an overconfident discriminator
- Balance G and D capacity- If D becomes too strong too fast, G's gradients vanish and training stalls
- Monitor both losses- A D loss near 0 usually signals D has overpowered G (or vice versa) -- losses should oscillate, not converge cleanly
- Use Wasserstein loss for stability- WGAN-GP largely avoids mode collapse and vanishing gradients compared to the vanilla minimax loss
WGAN-GP: Gradient Penalty Implementation
Enforces the 1-Lipschitz constraint on the critic by penalizing gradient norms away from 1, avoiding WGAN's brittle weight clipping.
python
def gradient_penalty(D, real, fake, device): batch_size = real.size(0) eps = torch.rand(batch_size, 1, 1, 1, device=device) interpolated = (eps * real + (1 - eps) * fake).requires_grad_(True) d_interpolated = D(interpolated) grads = torch.autograd.grad( outputs=d_interpolated, inputs=interpolated, grad_outputs=torch.ones_like(d_interpolated), create_graph=True, retain_graph=True )[0] grads = grads.view(batch_size, -1) gp = ((grads.norm(2, dim=1) - 1) ** 2).mean() return gp# critic loss = D(fake).mean() - D(real).mean() + lambda_gp * gradient_penalty(...)# lambda_gp is typically 10; the critic (not 'discriminator') has no sigmoid output
Evaluating GAN Output Quality
Loss values alone don't indicate sample quality -- use dedicated metrics.
- Frechet Inception Distance (FID)- Compares the mean/covariance of Inception-v3 features between real and generated image sets; lower is better, the most widely reported metric
- Inception Score (IS)- Measures both quality (confident class predictions) and diversity (varied class predictions) using a pretrained classifier; higher is better but less discriminative than FID
- Precision and Recall for generative models- Precision measures how much of the generated distribution overlaps real data (fidelity); recall measures how much of the real distribution is covered (diversity)
- LPIPS- Learned Perceptual Image Patch Similarity; used to measure diversity among generated samples or distance for tasks like image translation
- Kernel Inception Distance (KID)- An unbiased alternative to FID with a polynomial kernel, more reliable on small sample counts
- Human evaluation- Still the gold standard for perceptual realism; automated metrics correlate with but don't perfectly capture human judgment
Spectral Normalization for the Discriminator
Constrains each layer's spectral norm to 1, stabilizing training without the hard clipping of vanilla WGAN.
python
import torch.nn as nnimport torch.nn.utils.spectral_norm as spectral_normclass Discriminator(nn.Module): def __init__(self): super().__init__() self.net = nn.Sequential( spectral_norm(nn.Conv2d(3, 64, 4, 2, 1)), nn.LeakyReLU(0.2, inplace=True), spectral_norm(nn.Conv2d(64, 128, 4, 2, 1)), nn.LeakyReLU(0.2, inplace=True), spectral_norm(nn.Conv2d(128, 1, 4, 1, 0)), ) def forward(self, x): return self.net(x).view(-1)# SN-GAN: spectral norm alone (no gradient penalty) is often enough to stabilize training
Conditional GAN: Conditioning on Class Labels
Embed class labels and concatenate them into both the generator and discriminator inputs to control what gets generated.
python
class ConditionalGenerator(nn.Module): def __init__(self, latent_dim, num_classes, embed_dim=50): super().__init__() self.label_embed = nn.Embedding(num_classes, embed_dim) self.net = nn.Sequential( nn.Linear(latent_dim + embed_dim, 256), nn.ReLU(inplace=True), nn.Linear(256, 28 * 28), nn.Tanh(), ) def forward(self, z, labels): c = self.label_embed(labels) x = torch.cat([z, c], dim=1) return self.net(x).view(-1, 1, 28, 28)# Discriminator similarly concatenates the label embedding to its flattened input# or broadcasts it as an extra channel for convolutional discriminators
Advanced Stabilization Techniques
Techniques beyond basic label smoothing and loss monitoring for hard-to-train GANs.
- Two Time-Scale Update Rule (TTUR)- Use a lower learning rate for the generator than the discriminator (e.g. 1e-4 vs 4e-4), proven to converge to a local Nash equilibrium under mild assumptions
- EMA of generator weights- Maintain an exponential moving average of G's parameters and sample from the EMA copy at inference -- noticeably reduces artifacts versus the raw training weights
- Minibatch discrimination- Let the discriminator look at statistics across the whole minibatch, not just one sample, to directly penalize mode collapse
- Feature matching loss- Train G to match the discriminator's intermediate feature statistics on real vs. fake data instead of directly maximizing D's output
- PixelNorm- Normalizes feature vectors to unit length at each pixel in the generator (used in ProGAN/StyleGAN) instead of BatchNorm, avoiding signal magnitude escalation
- Progressive growing- Start training at low resolution (e.g. 4x4) and progressively add layers to increase resolution, stabilizing high-resolution image synthesis
Pro Tip
Falling discriminator loss to near zero doesn't mean training is going well -- it usually means the discriminator has 'won,' generator gradients are vanishing, and you need to slow D down or switch to a Wasserstein-style loss.
Was this cheat sheet helpful?
Explore Topics
#GenerativeAdversarialNetworksGANs#GenerativeAdversarialNetworksGANsCheatSheet#DataScience#Advanced#CoreConcepts#Minimal#GAN#Training#MachineLearning#CheatSheet#SkillVeris
