BLIP-2
BLIP-2 is a vision-language pretraining framework from Salesforce Research that connects frozen, pretrained image encoders and frozen large language models using a lightweight trainable module called the Q-Former, achieving strong…
Definition
BLIP-2 is a vision-language pretraining framework from Salesforce Research that connects frozen, pretrained image encoders and frozen large language models using a lightweight trainable module called the Q-Former, achieving strong multimodal performance while training far fewer parameters than end-to-end approaches.
Overview
BLIP-2, published in 2023 as a successor to BLIP, was designed to reduce the high computational cost of vision-language pretraining. Prior approaches, including BLIP itself, typically trained large image and text components jointly end-to-end, which required substantial compute and large paired datasets. BLIP-2 instead keeps both a frozen pretrained image encoder (such as a CLIP or EVA-CLIP vision transformer) and a frozen pretrained large language model (such as OPT or FlanT5) entirely unchanged, and introduces a compact trainable bridge called the Querying Transformer, or Q-Former. The Q-Former is a lightweight transformer that uses a fixed set of learnable query vectors to extract a fixed-length set of the most relevant visual features from the frozen image encoder's output, and then maps those features into a representation the frozen LLM can consume as if they were text tokens. Training happens in two stages: first the Q-Former is trained with vision-language contrastive, matching, and generative objectives to learn to extract useful visual representations; second, the Q-Former's output is connected to the frozen LLM and trained on image-to-text generation, teaching the LLM to interpret the visual query embeddings. Because only the small Q-Former (and a projection layer) is trained while the large vision and language backbones stay frozen, BLIP-2 dramatically reduces trainable parameters and compute cost compared to fully joint training, while achieving state-of-the-art results at the time on zero-shot image captioning and visual question answering. BLIP-2's Q-Former design became an influential building block, directly inspiring InstructBLIP and shaping the general pattern — reusing strong frozen unimodal models plus a small trainable connector — used by many subsequent vision-language models.
Key Concepts
- Connects frozen pretrained image encoders to frozen pretrained LLMs
- Introduces the Q-Former, a lightweight trainable querying transformer bridge
- Two-stage training: vision-language representation learning, then LLM alignment
- Dramatically fewer trainable parameters than end-to-end joint training
- Strong zero-shot image captioning and VQA performance
- Compatible with multiple LLM backbones (e.g., OPT, FlanT5)
- Architectural basis for InstructBLIP and many later connector-based VLMs