Idefics
Idefics is an open-source vision-language model family from Hugging Face that interleaves images and text as input to generate text output, built as an open reproduction of DeepMind's closed Flamingo model.
Definition
Idefics is an open-source vision-language model family from Hugging Face that interleaves images and text as input to generate text output, built as an open reproduction of DeepMind's closed Flamingo model.
Overview
Idefics (Image-aware Decoder Enhanced à la Flamingo with Interleaved Cross-attentionS) was created by Hugging Face and collaborators to give the open-source community a Flamingo-class multimodal model, since Flamingo itself was never released publicly. The original Idefics (available in 9B and 80B parameter variants) combined a frozen or lightly-tuned vision encoder with a large language model backbone, connected via cross-attention layers inserted throughout the LM, and trained on large interleaved image-text web corpora such as OBELICS. Idefics2, the successor, improved efficiency and accuracy substantially at a much smaller 8B parameter scale, adopting a more modern architecture that processes images at native resolution via a vision encoder (based on SigLIP) with a simpler MLP-based connector to the language model, and was trained on curated instruction and OCR-heavy data to improve document and chart understanding. Idefics3 pushed further using Llama 3.1 as the backbone and a larger, more diverse training mixture. The Idefics family matters historically because it was one of the first fully open (weights, training data references, and technical reports) large-scale vision-language models, at a time when comparable systems (GPT-4V, Flamingo, Gemini) were closed. This openness made it a reference point for researchers studying multimodal training at scale, and its interleaved image-text training approach — processing documents with figures, multi-image reasoning, and mixed-modality context — influenced later open VLM releases across the ecosystem.
Key Concepts
- Open reproduction of DeepMind's Flamingo architecture
- Interleaved image-and-text input handling for multi-image reasoning
- Cross-attention layers connecting a vision encoder to a language model backbone
- Idefics2 uses a SigLIP vision encoder and native-resolution image processing
- Idefics3 built on a Llama 3.1 language model backbone
- Trained partly on the open OBELICS interleaved web dataset
- Full open release of weights and technical reports
- Strong document, chart, and OCR-style understanding in later versions