Skip to main content
SAM segments whatever a point or a box points at, but it cannot find an object from a description. Grounding DINO has the opposite profile. It is an open-set detector that finds the objects named in free-form text and returns a box for each, but it produces no masks. Grounded SAM chains the two: the detector turns the text into boxes, and the boxes become SAM’s prompts Ren et al., 2024.

​
Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

Ren et al., arXiv:2401.14159

​
The pipeline

Given an image and a text prompt, Grounding DINO returns a box for every region that matches a phrase in the text. Each box is then passed to SAM as a box prompt, and SAM returns the mask inside it. The phrase that produced the box becomes the label of the mask, which SAM alone cannot provide.

​
Detect first, then segment

The authors split open-set segmentation into detection followed by segmentation because the two halves have very different data costs. Masks that match arbitrary text are expensive to annotate, so there is little training data for segmenting from text directly. Boxes are cheaper to annotate, so open-set detectors can be trained on much more data, and a detector only has to find coordinates, not pixels. Going from a box to a mask is then an easier problem than going from text to a mask, because the box already says where the object is. SAM solves that second problem well, having been trained on the 1.1 billion masks of SA-1B. On the Segmentation in the Wild (SGinW) benchmark, 25 datasets evaluated zero-shot, Grounding DINO-Base combined with SAM-Huge reaches 48.7 mean AP. The paper reports that this beats unified open-set segmentation models such as UNINEXT and OpenSeeD, and that replacing SAM with HQ-SAM, which produces sharper masks, improves the result further.

​
Assembling more models

The same idea extends to other expert models, as long as the output of one is a valid input to the next.
  • RAM-Grounded-SAM annotates images with no text at all. An image tagging model (Recognize Anything) or a captioning model (BLIP) produces the text, and Grounded SAM turns each tag into boxes and masks.
  • Grounded-SAM-SD edits images. The mask from Grounded SAM selects the region that a Stable Diffusion inpainting model repaints.
  • Grounded-SAM-OSX estimates 3D human pose. The boxes and masks of people feed a whole-body mesh recovery model.
Because the task is split into steps, you can inspect the output of each step and see which model made a mistake. The paper names this as an advantage over a single model trained end to end.

​
Limits of a fixed pipeline

The pipeline runs once, from left to right. The detector is the only component that reads the text, and it was trained to match phrases to regions. A request has to name the objects, as in “dog” or “red car”. A request that needs reasoning about the scene, such as “the object that protects you from the rain”, gives the detector no phrase to match. Nothing in the pipeline checks the masks either: if the detector draws the wrong box, SAM segments the wrong object. The paper’s closing section suggests the next step: a large language model that calls these models through their language interface. The SAM 3 Agent does this. A multimodal LLM plans, calls a segmentation model as a tool, looks at the result and tries again.
Found an error on this page? Report an issue.