Flamingo

Definition

Flamingo is a vision-language model from DeepMind that can process a mix of images, video, and text interleaved together and generate free-form text responses, using only a handful of example input-output pairs to adapt to a new task without additional training. It was an early and influential demonstration of few-shot in-context learning applied to multimodal, visual-and-text reasoning, extending a capability previously shown mainly in text-only large language models into the visual domain.

Overview

Flamingo was built to extend the few-shot learning capability that large language models like GPT-3 had demonstrated for text into the visual domain, where a model could be shown a small number of example image-text pairs within its input and then generalize to a new visual task without any gradient updates or fine-tuning. Before Flamingo, adapting a vision-language model to a new task typically meant fine-tuning it on task-specific labeled data; Flamingo's contribution was showing that, with the right architecture, a single pretrained model could instead be prompted with a few examples and immediately perform reasonably well on tasks like visual question answering or captioning it had not been explicitly trained for. Mechanically, Flamingo combines a pretrained vision encoder, which converts images or video frames into visual features, with a large pretrained language model, connecting the two through additional cross-attention layers inserted into the language model that let it attend to visual features at the relevant points while generating text. Because both the vision encoder and the base language model are largely kept frozen or lightly adapted, Flamingo's training focuses on learning the connecting layers that translate between visual and textual representations, which is more efficient than training a full multimodal model from scratch and allows the model to inherit strong language capabilities from the underlying language model. Flamingo sits alongside later open efforts like LLaVA and BLIP-2 as an early and influential example of connecting a vision encoder to a language model for generative multimodal output, and it differs from vision-language embedding models like CLIP, SigLIP, or ALIGN by producing free-form generated text conditioned on visual input rather than fixed embeddings for classification or retrieval. Its handling of interleaved sequences of images, video, and text was also distinctive relative to earlier vision-language models that typically accepted only a single image alongside a text prompt. In practice, Flamingo-style capabilities are used for visual question answering, image and video captioning, and rapid adaptation to new visual tasks through few-shot prompting rather than fine-tuning, which is valuable when labeled training data for a specific task is scarce. Its architecture and training approach influenced the design of subsequent open and proprietary multimodal models that also connect frozen or lightly adapted vision and language components. Limitations include that, as with other models relying on a frozen language model backbone, Flamingo's language generation quality and knowledge are bounded by the underlying language model it was built on, and its visual grounding, like other multimodal generative models, is subject to hallucination where generated text describes details not actually present in the image. Flamingo itself was a research system rather than a broadly released open product, so most practitioners today encounter its architectural ideas through the open models it influenced rather than using Flamingo directly.

Key Concepts

• Processes interleaved sequences of images, video, and text
• Few-shot adaptation to new visual tasks without fine-tuning
• Cross-attention layers connecting a frozen vision encoder and language model
• Generates free-form text conditioned on visual input
• Inherits language capability from an underlying pretrained language model
• Efficient training by keeping core components largely frozen
• Influential architecture for later vision-language model designs

Use Cases

Frequently Asked Questions

Related in AI Models