Chameleon (model)

Definition

Chameleon is a family of early-fusion multimodal foundation models developed by Meta AI (FAIR), described in a May 2024 research paper, designed to natively understand and generate interleaved sequences of text and images using a single unified token-based architecture rather than bolting a separate vision encoder onto a language model.

Overview

Most earlier multimodal models, including many contemporaneous vision-language models, use a "late fusion" design: a pretrained vision encoder (like CLIP or SigLIP) converts images into embeddings that are then fed into a separately pretrained language model. Chameleon instead uses "early fusion": images and text are both converted into discrete tokens from a shared vocabulary (using a learned image tokenizer/detokenizer) and processed by a single transformer trained from scratch on interleaved sequences of both modalities, allowing the model to generate mixed text-and-image output natively, not just accept image input. Meta's Chameleon paper detailed significant training stability challenges specific to early fusion at scale — mixing discrete image and text tokens in the same autoregressive sequence introduced optimization instabilities that standard techniques used for text-only LLMs didn't fully address — and described architectural and training modifications (such as adjustments to normalization) developed to stabilize training. The released models scaled up to 34 billion parameters, and Meta evaluated Chameleon on both text-only and mixed-modal benchmarks, reporting competitive performance on text tasks alongside strong results on tasks requiring interleaved image-text generation, such as generating a recipe with an accompanying illustrative image. Meta released a subset of Chameleon (7B and 34B models, with image generation capabilities disabled in the public release for safety reasons) under a research license, positioning the work as a step toward more general "any-to-any" multimodal models that can flexibly take in and produce combinations of text, images, and eventually other modalities within one architecture. Chameleon is frequently discussed alongside other early-fusion or unified multimodal efforts, such as Google DeepMind's Gemini (which also uses native multimodal training, albeit with different architectural choices) and later open efforts building on the early-fusion concept.

Key Concepts

• Early-fusion architecture: text and images share a single token vocabulary and transformer
• Natively generates interleaved text-and-image output, not just image-conditioned text
• Scales up to 34 billion parameters
• Introduced training stabilization techniques specific to mixed discrete-token sequences
• Evaluated on both text-only and mixed-modal generation benchmarks
• Public release excludes image generation for safety, offering understanding-only checkpoints
• Released under a research license by Meta AI (FAIR)

Use Cases

Frequently Asked Questions

Related in AI Models