> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Segment Anything

> How SAM turns a click or a box into a segmentation mask.

The Segment Anything Model (SAM) segments an object from a prompt: a point, a box, a rough mask or, in the paper's experiments, a short phrase of text [Kirillov et al., 2023](https://arxiv.org/abs/2304.02643). The text prompts were an exploratory experiment and are not part of the released model.

SAM does not assign a class to the mask, and the released model does not understand language. In practice it is paired with a model that does. A grounding detector or a vision-language model turns a request such as "the red mug" into a point or a box, and SAM turns that prompt into a mask. [Grounded SAM](https://aegean.ai/aiml-common/lectures/vlm/sam/grounded-sam/index.md) chains a detector with SAM in this way [Ren et al., 2024](https://arxiv.org/abs/2401.14159). LISA trains a multimodal LLM to produce the prompt for SAM's mask decoder [Lai et al., 2023](https://arxiv.org/abs/2308.00692). SAM 3 accepts short phrases directly, and the [SAM 3 Agent](https://aegean.ai/aiml-common/lectures/vlm/sam/agentic-sam/index.md) pairs it with a multimodal LLM for harder requests [Carion et al., 2025](https://arxiv.org/abs/2511.16719).

### [Segment Anything](https://arxiv.org/abs/2304.02643)

Kirillov et al., arXiv:2304.02643

## Architecture

SAM has three parts. A heavy image encoder runs once per image. A light prompt encoder and a light mask decoder then run once per prompt.

```mermaid theme={null}
graph TB
    img["Input image, 1024 x 1024"] --> enc["ViT image encoder, MAE pretrained"]
    enc --> emb["Image embedding, 64 x 64 x 256"]
    prm["Prompts: points, boxes, masks"] --> penc["Prompt encoder"]
    emb --> dec["Two-way transformer mask decoder"]
    penc --> dec
    dec --> out["Three candidate masks and an IoU score for each"]
```

This split makes SAM interactive. The image encoder takes most of the compute, and its output is reused for every prompt on the same image. Each new click only reruns the prompt encoder and the decoder, which the paper reports take about 50 ms in a web browser.

### Image encoder

The image encoder is a Vision Transformer (ViT-B, ViT-L or ViT-H) pretrained as a masked autoencoder (MAE). It takes the image resized to 1024 × 1024 and produces an embedding of 64 × 64 positions with 256 channels, one position for each 16 × 16 patch.

### Prompt encoder

Prompts come in two kinds. Sparse prompts are points and boxes. Each point, and each corner of a box, is represented by a positional encoding of its coordinates added to a learned embedding for its type: a foreground point, a background point, a top-left corner or a bottom-right corner. Text prompts in the paper go through the CLIP text encoder. Dense prompts are rough masks. These pass through a few convolutional layers and are added to the image embedding position by position.

### Mask decoder

The decoder is a small transformer with two layers. The prompt embeddings, plus learned output tokens, act as a set of tokens. In each layer the tokens attend to themselves, then the tokens attend to the image embedding, and then the image embedding attends back to the tokens. Because attention runs in both directions, the prompt information flows into the image features and the image features flow into the prompt tokens. After the two layers, two transposed convolutions upsample the image embedding by a factor of 4. Each output token goes through a small MLP that gives the weights of a linear classifier. The classifier is applied at every position of the upsampled embedding, and the result is the mask.

### Ambiguous prompts

One click often has more than one valid answer. A point on a shirt could mean the pattern on the shirt, the shirt, or the whole person. If SAM produced a single mask, training would push it toward the average of these answers, which matches none of them. SAM therefore predicts three masks for each prompt, roughly a subpart, a part and a whole object. During training only the mask with the lowest loss is backpropagated. A separate head predicts the IoU of each mask with the object, and this score ranks the three masks at inference time.

## Training data

SAM was trained on SA-1B, a dataset of 11 million images and 1.1 billion masks. The masks came from a data engine with three stages. In the assisted-manual stage, annotators labeled masks with an early SAM in the loop. In the semi-automatic stage, SAM proposed masks for the most confident objects and annotators added the objects it missed. In the fully automatic stage, SAM was prompted with a regular grid of points on each image, and the confident, stable masks were kept without annotation. Almost all of the 1.1 billion masks come from the last stage.

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/vlm/sam).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.