SAM 3: segmenting a concept
SAM and SAM 2 return one object per prompt. SAM 3 adds promptable concept segmentation. The prompt is a concept, given as a short noun phrase such as “yellow school bus”, as image exemplars, or as both, and the model returns a mask and an identity for every instance of the concept in an image or a video. The model has a detector for images and a tracker for video, and both share one vision-language backbone. The detector follows the DETR design. The image and the text prompt are encoded, a fusion encoder lets the image features cross-attend to the prompt tokens, and a decoder turns a set of learned object queries into boxes and masks. A separate presence token predicts only whether the concept appears in the image at all. Each object query then only has to localize its object, and its final score is its own score multiplied by the presence score. The paper reports that separating recognition from localization in this way improves detection, especially when training includes hard negative phrases that name things absent from the image. The tracker comes from SAM 2 and follows each detected object from frame to frame. SAM 3 is still limited to simple noun phrases. A request that describes an object by its relations to other objects, or that needs common sense, is not a noun phrase.The agent
In the SAM 3 Agent the multimodal LLM acts as planner and controller. It analyzes the image, makes a plan, calls tools, inspects the results and revises the plan, in a loop that ends when it is satisfied with the masks or concludes that no object matches. The LLM can call four tools:
After every intermediate call, the agent receives the state of its environment in two forms. The first is the input image with all current masks drawn on it, colored and numbered from 1 to N in decreasing order of SAM 3’s confidence, a format known as Set-of-Marks. The second is a short text message saying what changed, for example how many masks
segment_phrase produced. The LLM reasons over this image and message and chooses the next call.
Keeping the context short
Every step adds a new image to the conversation, and the paper reports that hard requests can take as many as 60 steps of trial and error. That history exceeds what even strong multimodal LLMs handle well. The agent therefore prunes it. Each time it callssegment_phrase, every intermediate step between the original request and that call is removed, along with the old masks. So that the LLM does not repeat failed attempts, it keeps a running list of every noun phrase it has already tried.
Results
The agent was evaluated zero-shot on ReasonSeg, a reasoning segmentation benchmark, and on OmniLabel. The agent was not trained on any referring-expression or reasoning segmentation data, and the paper reports that it still surpasses prior work on both. The same system prompt worked with every LLM tried, and the choice of LLM matters: on the ReasonSeg test set the score rises from 63.0 with Qwen2.5-VL 7B to 74.0 with Gemini 2.5 Pro. The failures show where the two halves of the system break. In one example from the paper, the request is “a black object that protects you from the rain, being held by a person in jeans”. The LLM correctly decides to segment “black umbrella”, and SAM 3 segments the umbrellas, but the LLM then selects the wrong mask. The paper attributes the error to the LLM’s visual reasoning, not to SAM 3.From pipeline to agent
The agent swaps the fixed pipeline for an LLM that chooses its next tool call from what it sees. That is the same structure as other tool-using agents: a model that can reason but not segment, a model that can segment but not reason, and a loop that connects them through an interface both understand, here a noun phrase going in and numbered masks coming back.
Found an error on this page? Report an issue.