Model Pruning

Definition

Model pruning is a technique that removes redundant or low-impact parameters from a trained neural network, reducing its size and computational cost while aiming to preserve accuracy.

Overview

Large neural networks are often over-parameterized, meaning many of their weights contribute little to the model's final predictions. Pruning identifies and removes these less important parameters — individual weights (unstructured pruning) or entire structural components like neurons, channels, or attention heads (structured pruning) — resulting in a smaller, sparser model that requires less memory and computation to run. Pruning is typically performed after training, and is often followed by a period of fine-tuning to help the remaining parameters compensate for the removed ones and recover any accuracy lost during the process. The tradeoff between how much is pruned and how much accuracy is preserved is a key design decision, and pruning is frequently applied iteratively, removing a small percentage of parameters at a time and fine-tuning between rounds. Alongside model quantization and model distillation, pruning is one of the standard techniques in the model-compression toolkit used to make large foundation models and other neural networks practical to deploy on constrained hardware, reduce inference costs, and lower the energy footprint of running AI systems at scale.

Key Concepts

• Removes redundant or low-impact weights from a trained neural network
• Unstructured pruning removes individual weights; structured pruning removes whole components
• Typically followed by fine-tuning to recover any lost accuracy
• Often applied iteratively across multiple rounds of pruning and fine-tuning
• Reduces model size, memory usage, and computational cost
• Complements quantization and distillation in model-compression pipelines

Use Cases

Frequently Asked Questions

Related in AI Fundamentals