Segment Anything Model
The Segment Anything Model (SAM) is a foundation model for image segmentation released by Meta AI Research in April 2023, designed to segment any object in an image given minimal prompts such as points, boxes, or text.
Definition
The Segment Anything Model (SAM) is a foundation model for image segmentation released by Meta AI Research in April 2023, designed to segment any object in an image given minimal prompts such as points, boxes, or text.
Overview
SAM was introduced by Meta AI along with the Segment Anything project, which included the model, a data engine for generating segmentation masks at scale, and the SA-1B dataset — over 1 billion segmentation masks across 11 million licensed, privacy-respecting images, the largest segmentation dataset released publicly at the time. The project's goal was to build a "foundation model" for segmentation, analogous to how large language models generalize across text tasks, so that a single pretrained model could segment objects in new images and domains it had never explicitly been trained on, without task-specific fine-tuning. SAM's architecture consists of three components: an image encoder (a Vision Transformer pretrained via masked autoencoding) that processes the image once into an embedding, a lightweight prompt encoder that converts user prompts (points, bounding boxes, rough masks, or text) into embeddings, and a fast mask decoder that combines these to predict segmentation masks in real time. Because the heavy image encoding happens once per image, SAM can respond to new prompts on the same image in milliseconds, making it practical for interactive use in image editing tools. SAM's promptable, zero-shot segmentation capability made it widely adopted across computer vision applications, from photo and video editing to medical imaging research, robotics, and augmented reality. Meta followed up with SAM 2 in 2024, which extended the same promptable segmentation approach to video, adding the ability to track segmented objects across frames. SAM's open release of both weights and the SA-1B dataset significantly accelerated research and product development in interactive and automated image segmentation.
Key Concepts
- Promptable segmentation using points, boxes, rough masks, or text
- Zero-shot generalization to new objects and image domains
- Vision Transformer image encoder combined with a lightweight prompt encoder and mask decoder
- Real-time mask prediction after a one-time image embedding step
- Trained on and released alongside the SA-1B dataset (over 1 billion masks)
- Open weights released by Meta AI Research
- Foundation for the video-capable follow-up model, SAM 2
Use Cases
Frequently Asked Questions
From the Blog
How AI Fraud Detection Systems Work
AI fraud detection spots suspicious transactions in real time by learning normal behavior and flagging anything that deviates, using both labeled examples and anomaly detection.
Read More AI & TechnologyShipping AI Features: Engineering Practices for LLM Products
Shipping an LLM feature is a delivery lifecycle, not a prompt. You scope the task so success is checkable, build an evaluation set before you tune anything, roll out behind a flag, instrument the funnel, and cap cost per user. This guide walks that lifecycle end to end and names the failure mode at each stage.
Read More AI & TechnologyHow to Add Source Citations to RAG Answers
Reliable citations come from threading a stable chunk identifier through retrieval into the prompt, asking for it back in a structured field, and then verifying the quoted span actually appears in that chunk. Anything less produces plausible references that point at the wrong document.
Read More AI & TechnologyHow to Store and Query Metadata Alongside Embeddings
Design the payload before you index anything: flat, typed, low-cardinality fields for the things you will filter on, with tenant and permission keys applied server-side on every query. Then decide deliberately between pre-filtering and post-filtering, because that choice determines whether restrictive filters return empty results.
Read More