> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agentic SAM

> A multimodal LLM plans, calls SAM 3 as a tool and checks the masks.

[Grounded SAM](https://aegean.ai/aiml-common/lectures/vlm/sam/grounded-sam/index.md) runs a fixed pipeline once: text to boxes, boxes to masks. It handles requests that name an object, such as "brown handbag", and fails on requests that need reasoning, such as "a black object that protects you from the rain, being held by a person in jeans". The SAM 3 paper handles the second kind with an agent [Carion et al., 2025](https://arxiv.org/abs/2511.16719). A multimodal LLM reads the request, decides which simple phrase to segment, calls SAM 3 with it, looks at the masks that come back and decides what to do next.

### [SAM 3: Segment Anything with Concepts](https://arxiv.org/abs/2511.16719)

Carion et al., arXiv:2511.16719

## SAM 3: segmenting a concept

[SAM](https://aegean.ai/aiml-common/lectures/vlm/sam/index.md) and SAM 2 return one object per prompt. SAM 3 adds promptable concept segmentation. The prompt is a concept, given as a short noun phrase such as "yellow school bus", as image exemplars, or as both, and the model returns a mask and an identity for every instance of the concept in an image or a video.

The model has a detector for images and a tracker for video, and both share one vision-language backbone. The detector follows the DETR design. The image and the text prompt are encoded, a fusion encoder lets the image features cross-attend to the prompt tokens, and a decoder turns a set of learned object queries into boxes and masks. A separate presence token predicts only whether the concept appears in the image at all. Each object query then only has to localize its object, and its final score is its own score multiplied by the presence score. The paper reports that separating recognition from localization in this way improves detection, especially when training includes hard negative phrases that name things absent from the image. The tracker comes from SAM 2 and follows each detected object from frame to frame.

SAM 3 is still limited to simple noun phrases. A request that describes an object by its relations to other objects, or that needs common sense, is not a noun phrase.

## The agent

In the SAM 3 Agent the multimodal LLM acts as planner and controller. It analyzes the image, makes a plan, calls tools, inspects the results and revises the plan, in a loop that ends when it is satisfied with the masks or concludes that no object matches.

```mermaid theme={null}
graph TB
    req["Image and user request"] --> llm["Multimodal LLM: plan the next action"]
    llm -->|"segment_phrase('black umbrella')"| sam["SAM 3"]
    sam --> state["Image with numbered masks, and a text message saying what changed"]
    llm -->|"examine_each_mask"| check["Render each mask on its own; keep the correct ones"]
    check --> state
    state --> llm
    llm -->|"select_masks_and_return([2])"| done["Final masks"]
    llm -->|"report_no_mask"| none["No matching object"]
```

The LLM can call four tools:

| Tool | What it does |
| - | - |
| `segment_phrase` | Calls SAM 3 with a short noun phrase. All previous masks are discarded, and the new masks are drawn on the image. |
| `examine_each_mask` | Renders each mask separately so the LLM can check small or overlapping masks one at a time and drop the wrong ones. |
| `select_masks_and_return` | Returns the chosen masks, by number, as the answer and ends the task. |
| `report_no_mask` | Reports that nothing in the image matches the request and ends the task. |

After every intermediate call, the agent receives the state of its environment in two forms. The first is the input image with all current masks drawn on it, colored and numbered from 1 to N in decreasing order of SAM 3's confidence, a format known as Set-of-Marks. The second is a short text message saying what changed, for example how many masks `segment_phrase` produced. The LLM reasons over this image and message and chooses the next call.

### Keeping the context short

Every step adds a new image to the conversation, and the paper reports that hard requests can take as many as 60 steps of trial and error. That history exceeds what even strong multimodal LLMs handle well. The agent therefore prunes it. Each time it calls `segment_phrase`, every intermediate step between the original request and that call is removed, along with the old masks. So that the LLM does not repeat failed attempts, it keeps a running list of every noun phrase it has already tried.

## Results

The agent was evaluated zero-shot on ReasonSeg, a reasoning segmentation benchmark, and on OmniLabel. The agent was not trained on any referring-expression or reasoning segmentation data, and the paper reports that it still surpasses prior work on both. The same system prompt worked with every LLM tried, and the choice of LLM matters: on the ReasonSeg test set the score rises from 63.0 with Qwen2.5-VL 7B to 74.0 with Gemini 2.5 Pro.

The failures show where the two halves of the system break. In one example from the paper, the request is "a black object that protects you from the rain, being held by a person in jeans". The LLM correctly decides to segment "black umbrella", and SAM 3 segments the umbrellas, but the LLM then selects the wrong mask. The paper attributes the error to the LLM's visual reasoning, not to SAM 3.

## From pipeline to agent

| | Grounded SAM | SAM 3 Agent |
| - | - | - |
| Who reads the request | An open-set detector | A multimodal LLM |
| Requests it handles | Phrases that name objects | Free-form requests that need reasoning |
| Control flow | One pass, left to right | A loop of tool calls until the LLM is satisfied |
| Checking the output | None | The LLM inspects the drawn masks before returning |
| Cost | One detector and one SAM call | Many LLM and SAM 3 calls, each adding an image to the context |

The agent swaps the fixed pipeline for an LLM that chooses its next tool call from what it sees. That is the same structure as other tool-using agents: a model that can reason but not segment, a model that can segment but not reason, and a loop that connects them through an interface both understand, here a noun phrase going in and numbered masks coming back.

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/vlm/sam/agentic-sam).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.