LoRA

Definition

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that adapts a large pretrained model by freezing its original weights and injecting small, trainable low-rank matrices into its layers. Only these added matrices — a tiny fraction of the model's total parameters — are trained, dramatically reducing the compute, memory, and storage needed to fine-tune large models while retaining performance close to full fine-tuning.

Overview

Full fine-tuning of a large language model requires updating every one of its potentially billions of parameters, which demands substantial GPU memory (to store gradients and optimizer states for all parameters) and produces a full-sized model copy for every fine-tuned variant. LoRA addresses this by exploiting the observation that the change in weights needed to adapt a pretrained model to a new task tends to have a low 'intrinsic rank' — meaning it can be well approximated by the product of two much smaller matrices. Instead of updating a weight matrix directly, LoRA freezes the original pretrained weights and adds a parallel path consisting of two small low-rank matrices whose product approximates the necessary weight update. During training, only these small matrices are updated via gradient descent; the original model weights never change. At inference time, the low-rank update can either be kept separate (allowing quick swapping between different LoRA adapters on the same base model) or merged back into the original weights for no added inference latency. This approach yields dramatic efficiency gains: LoRA can reduce the number of trainable parameters by orders of magnitude compared to full fine-tuning, correspondingly cutting GPU memory requirements and enabling fine-tuning of very large models on consumer-grade hardware. It also produces small adapter files (often megabytes rather than gigabytes), making it practical to store and distribute many task-specific or style-specific adapters for a single shared base model. LoRA is widely used in both language model and image generation (e.g., diffusion model) fine-tuning, and has inspired further variants like QLoRA (which combines LoRA with quantization of the base model for even greater memory savings) and other low-rank or sparse adaptation methods, collectively forming the core toolkit of modern parameter-efficient fine-tuning (PEFT).

Key Concepts

• Freezes the original pretrained model weights and adds small trainable low-rank matrices
• Reduces trainable parameters and memory requirements by orders of magnitude versus full fine-tuning
• Enables fine-tuning large models on consumer-grade GPUs
• Produces small, portable adapter files that can be swapped on a shared base model
• Can be merged into base weights for zero added inference latency, or kept separate for flexibility
• Widely used for both language models and image/diffusion model customization
• Basis for further techniques like QLoRA, which adds quantization for even greater efficiency
• Performance typically approaches, though may not always fully match, full fine-tuning

Use Cases

Frequently Asked Questions

Related in AI Fundamentals