Segment Anything Model
The Segment Anything Model (SAM) is a foundation model for image segmentation released by Meta AI Research in April 2023, designed to segment any object in an image given minimal prompts such as points, boxes, or text.
Definition
The Segment Anything Model (SAM) is a foundation model for image segmentation released by Meta AI Research in April 2023, designed to segment any object in an image given minimal prompts such as points, boxes, or text.
Overview
SAM was introduced by Meta AI along with the Segment Anything project, which included the model, a data engine for generating segmentation masks at scale, and the SA-1B dataset — over 1 billion segmentation masks across 11 million licensed, privacy-respecting images, the largest segmentation dataset released publicly at the time. The project's goal was to build a "foundation model" for segmentation, analogous to how large language models generalize across text tasks, so that a single pretrained model could segment objects in new images and domains it had never explicitly been trained on, without task-specific fine-tuning. SAM's architecture consists of three components: an image encoder (a Vision Transformer pretrained via masked autoencoding) that processes the image once into an embedding, a lightweight prompt encoder that converts user prompts (points, bounding boxes, rough masks, or text) into embeddings, and a fast mask decoder that combines these to predict segmentation masks in real time. Because the heavy image encoding happens once per image, SAM can respond to new prompts on the same image in milliseconds, making it practical for interactive use in image editing tools. SAM's promptable, zero-shot segmentation capability made it widely adopted across computer vision applications, from photo and video editing to medical imaging research, robotics, and augmented reality. Meta followed up with SAM 2 in 2024, which extended the same promptable segmentation approach to video, adding the ability to track segmented objects across frames. SAM's open release of both weights and the SA-1B dataset significantly accelerated research and product development in interactive and automated image segmentation.
Key Concepts
- Promptable segmentation using points, boxes, rough masks, or text
- Zero-shot generalization to new objects and image domains
- Vision Transformer image encoder combined with a lightweight prompt encoder and mask decoder
- Real-time mask prediction after a one-time image embedding step
- Trained on and released alongside the SA-1B dataset (over 1 billion masks)
- Open weights released by Meta AI Research
- Foundation for the video-capable follow-up model, SAM 2