> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Object Detection with Keyframe Gating and Vector Storage

> Building a pose-gated YOLOv8 + CLIP detection pipeline over Zenoh with pgvector storage for semantic mapping.

This page describes the methodology for building a keyframe-gated object detection pipeline that produces CLIP embeddings and stores them in a vector database. The pipeline runs alongside a ROS 2 simulated robot without depending on ROS 2 itself  - it uses Zenoh as the transport layer and PostgreSQL with pgvector for persistent storage.

The reference implementation is in the [turtlebot-maze](https://github.com/pantelis/turtlebot-maze) repository.

## Architecture

The detection pipeline sits outside the ROS 2 container. It subscribes to camera images and odometry via Zenoh, runs inference on GPU, and publishes enriched detections back to Zenoh. A separate ingest worker writes embeddings to pgvector.

```mermaid actions={true} theme={null}
graph TB
    subgraph ROS 2 Container
        GZ["Gazebo"] -->|sensor_msgs/Image| CAM["/camera/color/image_raw"]
        GZ -->|nav_msgs/Odometry| ODOM["/odom"]
    end

    subgraph Zenoh Transport
        ZB["zenoh-bridge-ros2dds"]
        ZR["Zenoh Router"]
        ZB <-->|peer| ZR
    end

    subgraph PyTorch Container
        DET["Detector<br/>(YOLOv8 + CLIP)"]
    end

    subgraph Ingest Worker
        ING["embedding_ingest.py"]
    end

    subgraph PostgreSQL
        PGV["pgvector<br/>detection_embeddings"]
    end

    CAM -->|CDR via DDS| ZB
    ODOM -->|CDR via DDS| ZB
    ZR -->|"camera/color/image_raw"| DET
    ZR -->|"odom"| DET
    DET -->|"tb/detections"| ZR
    ZR -->|"tb/detections"| ING
    ING -->|SQL INSERT| PGV

    classDef ros fill:#2e7d32,stroke:#1b5e20,color:#fff
    classDef zenoh fill:#0277bd,stroke:#01579b,color:#fff
    classDef gpu fill:#e65100,stroke:#bf360c,color:#fff
    classDef db fill:#37474f,stroke:#546e7a,color:#fff

    class GZ,CAM,ODOM ros
    class ZB,ZR zenoh
    class DET gpu
    class ING,PGV db
```

## CDR Deserialization

The zenoh-bridge-ros2dds forwards raw CDR (Common Data Representation) bytes from DDS to Zenoh without re-encoding. CDR is the binary serialization format used by DDS, the middleware layer beneath ROS 2. The detector uses the `pycdr2` Python library to deserialize these bytes directly into Python dataclasses  - no ROS 2 installation required.

Each ROS 2 message type maps to a pycdr2 dataclass:

```python theme={null}
from pycdr2 import IdlStruct
from pycdr2.types import uint8, uint32, int32, float64

@dataclass
class Image(IdlStruct, typename="sensor_msgs/msg/Image"):
    header: Header
    height: uint32
    width: uint32
    encoding: str
    is_bigendian: uint8
    step: uint32
    data: List[uint8]

@dataclass
class Odometry(IdlStruct, typename="nav_msgs/msg/Odometry"):
    header: Header
    child_frame_id: str
    pose: PoseWithCovariance
    twist: TwistWithCovariance
```

Deserialization in the callback:

```python theme={null}
img_msg = Image.deserialize(bytes(sample.payload))
odom_msg = Odometry.deserialize(bytes(sample.payload))
```

## Keyframe Gating

Not every camera frame warrants running inference. The camera streams at \~30 fps, but consecutive frames from a slow-moving robot are nearly identical. Keyframe gating selects frames worth processing based on how much the robot has moved since the last processed frame.

### Per-frame decision flow

1. A camera frame arrives (rate-limited to 10 Hz)
2. The detector checks the latest cached pose from the odometry subscriber
3. It computes the distance and angle change since the last accepted keyframe
4. If the robot has moved less than 0.5 m AND rotated less than 15 degrees, the frame is discarded  - no inference runs
5. If either threshold is exceeded, this frame becomes a keyframe  - YOLO and CLIP run

```python theme={null}
def is_keyframe() -> bool:
    dx = latest_pose[0] - last_keyframe_pose[0]
    dy = latest_pose[1] - last_keyframe_pose[1]
    dist = math.sqrt(dx * dx + dy * dy)
    dyaw = abs(latest_pose[2] - last_keyframe_pose[2])
    dyaw = abs(math.atan2(math.sin(dyaw), math.cos(dyaw)))
    if dist >= 0.5 or dyaw >= math.radians(15):
        last_keyframe_pose = latest_pose
        return True
    return False
```

When the robot is stationary, zero frames pass the gate  - no GPU compute is used. At typical navigation speeds (\~0.2 m/s), a keyframe fires roughly every 2-3 seconds.

## Detection and Embedding Extraction

Each keyframe goes through two models in sequence:

### YOLOv8 (detection)

YOLOv8 nano runs on the full frame and produces bounding boxes with class labels and confidence scores. The model is pretrained on COCO's 80 object classes.

```python theme={null}
results = model.predict(frame, conf=0.3, verbose=False)
```

### CLIP (embedding)

For each detection, the bounding box region is cropped from the original frame and passed through a CLIP image encoder (ViT-B/32, pretrained on LAION-2B). This produces a 512-dimensional embedding vector that captures the visual semantics of the detected object.

```python theme={null}
import open_clip

clip_model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-32", pretrained="laion2b_s34b_b79k"
)

# For each detection crop:
batch = torch.stack([preprocess(crop) for crop in crops]).to(device)
embeddings = clip_model.encode_image(batch)
embeddings = F.normalize(embeddings, dim=-1)  # L2-normalize
```

All crops from a single keyframe are batched into one forward pass for efficiency. The embeddings are L2-normalized so that cosine similarity reduces to a dot product  - this is what pgvector's vector operators expect.

### Why CLIP instead of YOLOv8 backbone features

YOLOv8's backbone produces spatial feature maps optimized for detection, not for cross-instance visual similarity. CLIP embeddings are trained on image-text pairs across millions of concepts, making them effective for:

* Matching the same physical object seen from different angles
* Comparing objects across different runs (Run A mapping vs Run B re-localization)
* KNN search in pgvector for semantic re-localization

## Zenoh Payload Format

The detector publishes a JSON envelope to the `tb/detections` Zenoh key for each keyframe:

```json theme={null}
{
  "keyframe_id": 42,
  "timestamp": 1711281234.567,
  "map_x": 2.14,
  "map_y": 0.83,
  "map_yaw": 0.785,
  "detections": [
    {
      "class": "cup",
      "confidence": 0.87,
      "bbox": [120.0, 80.0, 250.0, 310.0],
      "embedding": "<base64-encoded 512 x float32>",
      "embedding_dim": 512,
      "embedding_model": "ViT-B-32/laion2b_s34b_b79k"
    }
  ]
}
```

The embedding field is a base64-encoded byte string of 512 little-endian float32 values (2048 bytes raw). The ingest worker decodes this before writing to pgvector.

## Vector Database Storage

An ingest worker subscribes to `tb/detections` on the Zenoh router and writes each detection with its embedding and pose metadata to PostgreSQL with pgvector.

### Schema

```sql theme={null}
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE detection_embeddings (
  det_pk bigserial PRIMARY KEY,
  keyframe_id integer,
  class_name text NOT NULL,
  confidence real NOT NULL,
  bbox real[] NOT NULL,
  map_x real,
  map_y real,
  map_yaw real,
  embedding_model text,
  embedding vector(512),
  ingested_at timestamptz NOT NULL DEFAULT now()
);
```

### Querying similar objects

Once embeddings are stored, finding visually similar detections is a single SQL query:

```sql theme={null}
SELECT class_name, confidence, map_x, map_y,
       embedding <=> '[0.012, -0.034, ...]'::vector AS distance
FROM detection_embeddings
ORDER BY distance
LIMIT 5;
```

The `<=>` operator computes cosine distance. Because the embeddings are L2-normalized, this is equivalent to `1 - dot_product`.

## Docker Services

The pipeline is defined in `docker-compose.yaml` with these services:

| Service               | Role                                                    |
| --------------------- | ------------------------------------------------------- |
| `demo-world-enhanced` | Gazebo simulation with Nav2                             |
| `zenoh-router`        | Zenoh message bus with in-memory storage                |
| `zenoh-bridge`        | DDS to Zenoh bridge (forwards camera, odom, detections) |
| `detector`            | YOLOv8 + CLIP keyframe detector (GPU)                   |
| `embedding-ingest`    | Zenoh subscriber that writes to pgvector                |
| `vector`              | PostgreSQL with pgvector extension                      |
| `detection-logger`    | Appends raw detections to JSONL files                   |

To bring up the full pipeline:

```bash theme={null}
docker compose up -d \
  demo-world-enhanced \
  zenoh-router zenoh-bridge \
  detector embedding-ingest \
  vector detection-logger
```

***

<Callout icon="pen-to-square" iconType="regular">
  [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/simulation/object-detection.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).
</Callout>