Object detection and semantic localization in the house world

Object detectors localize objects as bounding boxes in an image. A semantic map instead represents physical objects at locations in a fixed coordinate frame. Depth measurements provide the geometric information needed to back-project image coordinates into three-dimensional camera coordinates and, subsequently, into the map frame. In this assignment, a simulated TurtleBot traverses a furnished house and constructs a semantic database indexed by object class. The completed system must report the location of each object instance and the robot poses from which it was observed. Each physical object is represented by one database record, and each observation stores the robot pose, camera pose, bounding box, confidence score, and visual embedding used to estimate that record.

Bringing up the house

From the course repository, start the house simulation, Zenoh router, and ROS 2 bridge:

docker compose up demo-world-house zenoh-router zenoh-bridge

The demo-world-house service launches the AWS RoboMaker residential environment in Gazebo together with Nav2. The environment is derived from the open-source small-house assets and contains 68 models whose names begin with aws_robomaker_residential_*. These include beds, chairs, sofas, coffee tables, air conditioners, carpets, curtains, chandeliers, night stands, wardrobes, and desk portraits. The TurtleBot begins near the entrance. A precomputed occupancy grid with a resolution of 5 cm per pixel is provided at tb_worlds/maps/house_world_map.yaml. The Gazebo worlds page describes the environment, its models, and the procedure for adding models. All Compose services use host networking. Run only one world service at a time; otherwise, the Gazebo instances contend for the same ports and prevent the Nav2 stack from operating correctly. Traverse the environment by teleoperation or with Nav2 waypoints, and record a systematic room-by-room route. The evaluation includes spatial coverage, so the route must extend beyond the entrance area. The remaining Compose services transport simulator data to persistent storage. zenoh-router provides the message bus and in-memory storage. zenoh-bridge runs zenoh-bridge-ros2dds and forwards camera, depth, odometry, and detection messages. The GPU-enabled detector service contains the reference YOLOv8 and CLIP pipeline. embedding-ingest subscribes to Zenoh and writes embeddings to vector, a PostgreSQL database with pgvector exposed on port 5436. A second PostgreSQL service, age, provides Apache AGE on port 5435.

Task 1: Detector node

Zenoh decouples the perception pipeline from the ROS 2 installation. The detector runs outside the ROS 2 container as a standard Python process and does not import rclpy. It can therefore process robot data on a GPU host without a local ROS installation. The pipeline uses the following Zenoh keys:

Zenoh key	ROS topic	Message
`camera/color/image_raw`	`/camera/color/image_raw`	`sensor_msgs/Image`, RGB
`camera/depth/image_rect_raw`	`/camera/depth/image_rect_raw`	`sensor_msgs/Image`, float32 meters
`odom`	`/odom`	`nav_msgs/Odometry`
`tb/detections`	Published by the detector	JSON envelope

The bridge forwards the raw Common Data Representation (CDR) payload from DDS without re-encoding it. Define the required ROS message types as IdlStruct dataclasses and deserialize each Zenoh payload with pycdr2. The following abbreviated definition illustrates the required structure:

from dataclasses import dataclass

from pycdr2 import IdlStruct
from pycdr2.types import sequence, uint8, uint32


@dataclass
class Time(IdlStruct, typename="builtin_interfaces::msg::dds_::Time_"):
    sec: int
    nanosec: uint32


@dataclass
class Header(IdlStruct, typename="std_msgs::msg::dds_::Header_"):
    stamp: Time
    frame_id: str


@dataclass
class Image(IdlStruct, typename="sensor_msgs::msg::dds_::Image_"):
    header: Header
    height: uint32
    width: uint32
    encoding: str
    is_bigendian: uint8
    step: uint32
    data: sequence[uint8]


image = Image.deserialize(bytes(sample.payload))

Limit the incoming color stream to 10 Hz. Accept a frame as a keyframe only when the robot has translated or rotated sufficiently relative to the last accepted pose:

import math


def wrapped_angle_difference(a: float, b: float) -> float:
    return abs(math.atan2(math.sin(a - b), math.cos(a - b)))


def accept_keyframe(x, y, yaw, previous_pose) -> bool:
    if previous_pose is None:
        return True

    old_x, old_y, old_yaw = previous_pose
    translation = math.hypot(x - old_x, y - old_y)
    rotation = wrapped_angle_difference(yaw, old_yaw)

    return translation >= 0.5 or rotation >= math.radians(15)

Update the stored reference pose only after accepting a keyframe. Under this policy, a stationary robot produces no keyframes and requires no detector inference. At a nominal speed of 0.2 m/s, the translation threshold is reached approximately every two to three seconds. Apply YOLOv8 nano, pretrained on the 80 COCO classes, with a confidence threshold of 0.3. Crop each accepted detection from the color image. Process all crops from a keyframe as one batch with the open_clip ViT-B-32 model and the laion2b_s34b_b79k weights. L2-normalize the resulting 512-dimensional embeddings so that their dot product is equal to cosine similarity. CLIP embeddings are used in place of YOLOv8 backbone features because they are more suitable for associating an object across changes in viewpoint. The detection and embedding pipeline describes these stages in more detail. Study object_detector.py in the reference repository and explain its data flow in the report. The submitted implementation must demonstrate independent work; an unchanged copy of the reference file does not satisfy this requirement. YOLOv8 was trained on photographs, whereas the simulated environment is rendered from AWS furniture meshes. In this environment, the most consistently detected classes are chair, couch, bed, tv, potted plant, dining table, vase, book, clock, refrigerator, sink, toilet, microwave, and oven. Many other COCO classes are not represented. Quantify this domain and class-coverage limitation with per-class measurements. Additional COCO objects, such as a cup, bottle, or laptop, may be placed on a kitchen counter by following the model-placement procedure on the Gazebo worlds page.

Task 2: From bounding box to map coordinates

Subscribe to the depth and odometry streams in addition to the color stream. For each accepted color keyframe, select the depth image and odometry sample with the nearest timestamps. Reject the keyframe if either timestamp differs from the color timestamp by more than 200 ms. Arrival order does not provide synchronization because transport and inference delays can reorder messages. Gazebo publishes depth images as 32-bit floating-point values in meters. For each detection, reduce the bounding box to a documented central fraction, collect the finite positive depth samples in that region, and use their median as the object depth. A single center pixel may lie between chair slats or legs and measure the wall behind the object. The median over an interior region is less sensitive to such gaps, object boundaries, and isolated invalid samples. Reject a detection when the interior region contains too few valid samples. Let

(u,v)

be the center of the detection box and let

d

be its median depth. The simulated Intel RealSense D435i produces images at

320 \times 240

pixels with

f_x = f_y = 277.13,\qquad c_x = 160,\qquad c_y = 120.

Back-project the detection into the camera optical frame:

x_o = \frac{(u-c_x)d}{f_x},\qquad y_o = \frac{(v-c_y)d}{f_y},\qquad z_o = d.

The optical axes are

z

-forward,

x

-right, and

y

-down. ROS body axes are

x

-forward,

y

-left, and

z

-up, giving the fixed optical-to-body conversion

\begin{bmatrix} x_b\\ y_b\\ z_b \end{bmatrix} = \begin{bmatrix} 0 & 0 & 1\\ -1 & 0 & 0\\ 0 & -1 & 0 \end{bmatrix} \begin{bmatrix} x_o\\ y_o\\ z_o \end{bmatrix}.

The camera transform from the URDF has translation

t_{bc} = \begin{bmatrix} 0.064\\ -0.065\\ 0.094 \end{bmatrix} \text{ meters}

and no rotation. Compose the full transform in this order:

p_{\text{map}} = T_{mb} \, T_{bc} \, R_{co} \, p_{\text{optical}}

Here

R_{co}

is the optical-to-body rotation,

T_{bc}

the fixed base_link to camera transform, and

T_{mb}

the map to base_link transform. Implement the chain with homogeneous coordinates and construct

T_{mb}

from the synchronized robot pose. Check the odometry message frame before treating that pose as a map pose. When it is expressed in an odometry frame, apply the corresponding map-to-odometry transform first and document how you obtained it. An incorrect optical-frame conversion may produce a map that appears plausible but is mirrored or rotated by 90 degrees. Test the complete transform chain against a hand-computed result. The test must pass a known robot pose, bounding-box pixel, and depth through the implementation and compare the resulting map coordinate with the expected value using a stated tolerance. Select a pixel away from the image center so that the test exercises every axis conversion.

Task 3: Semantic object database

Create the pgvector extension and use these two tables:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE semantic_objects (
    object_id BIGSERIAL PRIMARY KEY,
    class_name TEXT NOT NULL,
    map_x DOUBLE PRECISION NOT NULL,
    map_y DOUBLE PRECISION NOT NULL,
    map_z DOUBLE PRECISION NOT NULL,
    observation_count INTEGER NOT NULL CHECK (observation_count > 0),
    mean_embedding vector(512) NOT NULL,
    first_seen_ns BIGINT NOT NULL,
    last_seen_ns BIGINT NOT NULL
);

CREATE INDEX semantic_objects_class_name_idx
    ON semantic_objects (class_name);

CREATE TABLE object_observations (
    observation_id BIGSERIAL PRIMARY KEY,
    object_id BIGINT NOT NULL
        REFERENCES semantic_objects (object_id)
        ON DELETE CASCADE,
    seen_at_ns BIGINT NOT NULL,

    map_x DOUBLE PRECISION NOT NULL,
    map_y DOUBLE PRECISION NOT NULL,
    map_yaw DOUBLE PRECISION NOT NULL,

    camera_x DOUBLE PRECISION NOT NULL,
    camera_y DOUBLE PRECISION NOT NULL,
    camera_z DOUBLE PRECISION NOT NULL,
    camera_qx DOUBLE PRECISION NOT NULL,
    camera_qy DOUBLE PRECISION NOT NULL,
    camera_qz DOUBLE PRECISION NOT NULL,
    camera_qw DOUBLE PRECISION NOT NULL,

    bearing_rad DOUBLE PRECISION NOT NULL,
    range_m DOUBLE PRECISION NOT NULL,

    bbox_xmin INTEGER NOT NULL,
    bbox_ymin INTEGER NOT NULL,
    bbox_xmax INTEGER NOT NULL,
    bbox_ymax INTEGER NOT NULL,
    confidence REAL NOT NULL,
    embedding vector(512) NOT NULL
);

CREATE INDEX object_observations_object_id_idx
    ON object_observations (object_id);

Use ROS time in nanoseconds for all timestamp columns, and express the stored robot pose in the map frame. Derive the camera position and orientation from the localization transform chain. The report must state whether range_m denotes horizontal or three-dimensional distance and must define the zero direction and positive sign convention for bearing_rad. Each row in semantic_objects represents the current estimate of one physical object. The corresponding rows in object_observations preserve the measurements used to form that estimate. This separation supports indexed retrieval of every pose from which an object was observed without requiring a scan of the raw detector output. The same database supplies the semantic graph layer, in which objects and spatial relations are represented as graph entities.

Task 4: Deduplication

Associate each new detection only with stored objects that have the same class_name and whose fused map positions lie within a selected spatial radius. If the candidate set is empty, create a semantic object and its first observation. If it contains one object, merge the detection with that object. If it contains multiple objects, select the candidate with the highest CLIP cosine similarity. Insert a new observation row for every accepted detection. Select the association radius using measurements of depth noise, transform error, bounding-box variation, and viewpoint change. Document both the selected value and the measurements used to justify it. On a merge, update the fused position with a running average:

\bar{p}_{n+1} = \frac{n\bar{p}_n + p_{n+1}}{n+1}.

Update the mean embedding by the same rule, L2-normalize it again, increment observation_count, update last_seen_ns, and insert the new evidence row. In pgvector, <=> is cosine distance. For normalized vectors, it equals one minus their dot product, so the smallest distance identifies the most similar candidate. A group of four dining chairs illustrates the limitation of spatial-radius association. Although the chairs are distinct objects, several may fall within a radius large enough to accommodate the position error obtained when one chair is viewed from opposite sides. A smaller radius tends to produce duplicate records, whereas a larger radius tends to produce false merges. Measure how the selected radius and the embedding-based tie-break rule affect both outcomes.

Task 5: Querying the database

Implement and demonstrate the following parameterized queries. Do not interpolate values into SQL. Find the nearest instance of a requested class to a map point:

SELECT
    object_id,
    class_name,
    map_x,
    map_y,
    map_z,
    sqrt(
        power(map_x - $2, 2) +
        power(map_y - $3, 2)
    ) AS distance
FROM semantic_objects
WHERE class_name = $1
ORDER BY distance
LIMIT 1;

List every object inside a rectangular map region:

SELECT
    object_id,
    class_name,
    map_x,
    map_y,
    map_z,
    observation_count
FROM semantic_objects
WHERE map_x BETWEEN $1 AND $2
  AND map_y BETWEEN $3 AND $4
ORDER BY class_name, object_id;

Produce a histogram of stored objects per class:

SELECT
    class_name,
    count(*) AS object_count
FROM semantic_objects
GROUP BY class_name
ORDER BY object_count DESC, class_name;

List every robot pose from which one object was seen:

SELECT
    observation_id,
    seen_at_ns,
    map_x,
    map_y,
    map_yaw,
    camera_x,
    camera_y,
    camera_z,
    bearing_rad,
    range_m,
    confidence
FROM object_observations
WHERE object_id = $1
ORDER BY seen_at_ns;

Evaluation

Use the world SDF to establish ground-truth instance counts for every detectable class. Count distinct physical instances separately, including instances that use similar mesh assets or model names. Define instance precision as the number of correctly matched stored objects divided by the total number of stored objects. Define instance recall as the number of correctly matched stored objects divided by the number of ground-truth objects. The duplicate rate is the number of stored objects divided by the number of ground-truth objects. Its ideal value is 1.0; values above 1.0 indicate duplicate records, while values below 1.0 may indicate missed objects or false merges. The detection rate is the fraction of relevant keyframes that contain at least one correct detection of the class, where a relevant keyframe is one in which an instance of that class is visible. Also report false merges, in which one stored record contains observations from multiple physical objects, and misses, in which a ground-truth object has no stored record. Interpret these measures using the observation histories and estimated map positions. The chair row is an example. Add one row for each remaining class.

Class	True instances	Stored objects	Correct matches	Precision	Recall	Duplicate rate	Detection rate	False merges
chair	8	9	7	0.778	0.875	1.125	0.840	1

Include a map plot of the fused object positions, with each point labeled by class and object identifier. Overlay the traversal route or the accepted robot poses to distinguish incomplete spatial coverage from detector failures. Discuss classes that are absent from the environment, rarely visible, or consistently missed by the detector.

Deliverables

Submit:

The detector source code, including Zenoh subscriptions, CDR message definitions, keyframe gating, YOLOv8 inference, and batched CLIP embedding.
The depth synchronization and map back-projection implementation.
The transform-chain unit test and its hand-computed expected result.
The SQL schema and database-ingest code.
The deduplication implementation with its documented radius and similarity rule.
Executable versions of the four required database queries.
A database export containing semantic objects and their observations.
A map plot showing object estimates, identifiers, and coverage.
A report containing the route, per-class evaluation, duplicate-rate analysis, failure cases, and design choices.
A short README with commands needed to run your implementation against the compose services.

Grading

Component	Points
Detector node	20
Back-projection with unit test	25
Schema and ingest	20
Deduplication with measured duplicate rate	25
Report	10
Total	100

Edit this page on GitHub or file an issue.

Course

Assignments

Study Guides

Object detection and semantic localization in the house world

Bringing up the house

Task 1: Detector node

Task 2: From bounding box to map coordinates

Task 3: Semantic object database

Task 4: Deduplication

Task 5: Querying the database

Evaluation

Deliverables

Grading

​Bringing up the house

​Task 1: Detector node

​Task 2: From bounding box to map coordinates

​Task 3: Semantic object database

​Task 4: Deduplication

​Task 5: Querying the database

​Evaluation

​Deliverables

​Grading

Bringing up the house

Task 1: Detector node

Task 2: From bounding box to map coordinates

Task 3: Semantic object database

Task 4: Deduplication

Task 5: Querying the database

Evaluation

Deliverables

Grading