Transfer Learning Cheat Sheet

Covers feature extraction versus fine-tuning, freezing layers, and practical PyTorch code for adapting a pretrained model to a new task.

2 PagesIntermediateMar 8, 2026

Core Concepts

The vocabulary of adapting pretrained models.

Feature extraction- Freeze the pretrained backbone entirely and train only a new head on top of its fixed features
Fine-tuning- Unfreeze some or all pretrained layers and continue training them, usually at a lower learning rate
Frozen layer- A layer whose parameters are excluded from gradient updates (requires_grad = False)
Domain shift- When the target task's data distribution differs meaningfully from the pretraining data, requiring more unfreezing
Discriminative learning rates- Using smaller learning rates for earlier (more general) layers and larger rates for later (more task-specific) layers

Feature Extraction with a Frozen Backbone

Freeze a pretrained CNN and train only a new classification head.

python

import torch.nn as nnfrom torchvision import modelsmodel = models.resnet50(weights='IMAGENET1K_V2')for param in model.parameters():    param.requires_grad = False        # freeze everything# Replace the final layer - new params are trainable by defaultmodel.fc = nn.Linear(model.fc.in_features, num_classes)optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)

Fine-Tuning the Last Few Layers

Gradually unfreeze layers closest to the output for a more task-specific adaptation.

python

# Unfreeze just layer4 and the classifier headfor name, param in model.named_parameters():    param.requires_grad = name.startswith('layer4') or name.startswith('fc')optimizer = torch.optim.Adam([    {'params': model.layer4.parameters(), 'lr': 1e-5},  # smaller lr for pretrained layers    {'params': model.fc.parameters(), 'lr': 1e-3},       # larger lr for new head])

Choosing a Strategy

How much of the model to adapt.

Small dataset, similar domain- Feature extraction (freeze everything) usually works best and avoids overfitting
Large dataset, similar domain- Fine-tune the whole network at a low learning rate
Small dataset, different domain- Fine-tune only the last few layers; early layers capture generic features (edges, textures) that transfer well
Large dataset, different domain- Fine-tune the whole network, or train from scratch if the domain gap is extreme

Pro Tip

Use a much smaller learning rate for unfrozen pretrained layers than for a newly initialized head -- a single high learning rate applied to both can quickly destroy useful pretrained weights ('catastrophic forgetting').

Was this cheat sheet helpful?

Batch Normalization Cheat Sheet

Generative Adversarial Networks (GANs) Cheat Sheet