> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # Object detection and semantic localization in the house world > Drive a TurtleBot through the Gazebo house world, back-project COCO detections into the map frame, and build a deduplicated semantic object database indexed by class. Object detectors localize objects as bounding boxes in an image. A semantic map instead represents physical objects at locations in a fixed coordinate frame. Depth measurements provide the geometric information needed to back-project image coordinates into three-dimensional camera coordinates and, subsequently, into the map frame. In this assignment, a simulated TurtleBot traverses a furnished house and constructs a semantic database indexed by object class. The completed system must report the location of each object instance and the robot poses from which it was observed. Each physical object is represented by one database record, and each observation stores the robot pose, camera pose, bounding box, confidence score, and visual embedding used to estimate that record. ## Bringing up the house From the [course repository](https://github.com/pantelis/turtlebot-maze), start the house simulation, Zenoh router, and ROS 2 bridge: ```bash theme={null} docker compose up demo-world-house zenoh-router zenoh-bridge ``` The `demo-world-house` service launches the AWS RoboMaker residential environment in Gazebo together with Nav2. The environment is derived from the open-source small-house assets and contains 68 models whose names begin with `aws_robomaker_residential_*`. These include beds, chairs, sofas, coffee tables, air conditioners, carpets, curtains, chandeliers, night stands, wardrobes, and desk portraits. The TurtleBot begins near the entrance. A precomputed occupancy grid with a resolution of 5 cm per pixel is provided at `tb_worlds/maps/house_world_map.yaml`. The [Gazebo worlds page](/aiml-common/lectures/simulation/gazebo-worlds) describes the environment, its models, and the procedure for adding models. All Compose services use host networking. Run only one world service at a time; otherwise, the Gazebo instances contend for the same ports and prevent the Nav2 stack from operating correctly. Traverse the environment by teleoperation or with Nav2 waypoints, and record a systematic room-by-room route. The evaluation includes spatial coverage, so the route must extend beyond the entrance area. The remaining Compose services transport simulator data to persistent storage. `zenoh-router` provides the message bus and in-memory storage. `zenoh-bridge` runs `zenoh-bridge-ros2dds` and forwards camera, depth, odometry, and detection messages. The GPU-enabled `detector` service contains the reference YOLOv8 and CLIP pipeline. `embedding-ingest` subscribes to Zenoh and writes embeddings to `vector`, a PostgreSQL database with pgvector exposed on port 5436. A second PostgreSQL service, `age`, provides Apache AGE on port 5435. ## Task 1: Detector node Zenoh decouples the perception pipeline from the ROS 2 installation. The detector runs outside the ROS 2 container as a standard Python process and does not import `rclpy`. It can therefore process robot data on a GPU host without a local ROS installation. The pipeline uses the following Zenoh keys: | Zenoh key | ROS topic | Message | | ----------------------------- | ------------------------------ | ----------------------------------- | | `camera/color/image_raw` | `/camera/color/image_raw` | `sensor_msgs/Image`, RGB | | `camera/depth/image_rect_raw` | `/camera/depth/image_rect_raw` | `sensor_msgs/Image`, float32 meters | | `odom` | `/odom` | `nav_msgs/Odometry` | | `tb/detections` | Published by the detector | JSON envelope | The bridge forwards the raw Common Data Representation (CDR) payload from DDS without re-encoding it. Define the required ROS message types as `IdlStruct` dataclasses and deserialize each Zenoh payload with `pycdr2`. The following abbreviated definition illustrates the required structure: ```python theme={null} from dataclasses import dataclass from pycdr2 import IdlStruct from pycdr2.types import sequence, uint8, uint32 @dataclass class Time(IdlStruct, typename="builtin_interfaces::msg::dds_::Time_"): sec: int nanosec: uint32 @dataclass class Header(IdlStruct, typename="std_msgs::msg::dds_::Header_"): stamp: Time frame_id: str @dataclass class Image(IdlStruct, typename="sensor_msgs::msg::dds_::Image_"): header: Header height: uint32 width: uint32 encoding: str is_bigendian: uint8 step: uint32 data: sequence[uint8] image = Image.deserialize(bytes(sample.payload)) ``` Limit the incoming color stream to 10 Hz. Accept a frame as a *keyframe* only when the robot has translated or rotated sufficiently relative to the last accepted pose: ```python theme={null} import math def wrapped_angle_difference(a: float, b: float) -> float: return abs(math.atan2(math.sin(a - b), math.cos(a - b))) def accept_keyframe(x, y, yaw, previous_pose) -> bool: if previous_pose is None: return True old_x, old_y, old_yaw = previous_pose translation = math.hypot(x - old_x, y - old_y) rotation = wrapped_angle_difference(yaw, old_yaw) return translation >= 0.5 or rotation >= math.radians(15) ``` Update the stored reference pose only after accepting a keyframe. Under this policy, a stationary robot produces no keyframes and requires no detector inference. At a nominal speed of 0.2 m/s, the translation threshold is reached approximately every two to three seconds. Apply YOLOv8 nano, pretrained on the 80 COCO classes, with a confidence threshold of 0.3. Crop each accepted detection from the color image. Process all crops from a keyframe as one batch with the `open_clip` ViT-B-32 model and the `laion2b_s34b_b79k` weights. L2-normalize the resulting 512-dimensional embeddings so that their dot product is equal to cosine similarity. CLIP embeddings are used in place of YOLOv8 backbone features because they are more suitable for associating an object across changes in viewpoint. The [detection and embedding pipeline](/aiml-common/lectures/simulation/object-detection) describes these stages in more detail. Study `object_detector.py` in the reference repository and explain its data flow in the report. The submitted implementation must demonstrate independent work; an unchanged copy of the reference file does not satisfy this requirement. YOLOv8 was trained on photographs, whereas the simulated environment is rendered from AWS furniture meshes. In this environment, the most consistently detected classes are chair, couch, bed, tv, potted plant, dining table, vase, book, clock, refrigerator, sink, toilet, microwave, and oven. Many other COCO classes are not represented. Quantify this domain and class-coverage limitation with per-class measurements. Additional COCO objects, such as a cup, bottle, or laptop, may be placed on a kitchen counter by following the model-placement procedure on the Gazebo worlds page. ## Task 2: From bounding box to map coordinates Subscribe to the depth and odometry streams in addition to the color stream. For each accepted color keyframe, select the depth image and odometry sample with the nearest timestamps. Reject the keyframe if either timestamp differs from the color timestamp by more than 200 ms. Arrival order does not provide synchronization because transport and inference delays can reorder messages. Gazebo publishes depth images as 32-bit floating-point values in meters. For each detection, reduce the bounding box to a documented central fraction, collect the finite positive depth samples in that region, and use their median as the object depth. A single center pixel may lie between chair slats or legs and measure the wall behind the object. The median over an interior region is less sensitive to such gaps, object boundaries, and isolated invalid samples. Reject a detection when the interior region contains too few valid samples. Let $(u,v)$ be the center of the detection box and let $d$ be its median depth. The simulated Intel RealSense D435i produces images at $320 \times 240$ pixels with $$ f_x = f_y = 277.13,\qquad c_x = 160,\qquad c_y = 120. $$ Back-project the detection into the camera optical frame: $$ x_o = \frac{(u-c_x)d}{f_x},\qquad y_o = \frac{(v-c_y)d}{f_y},\qquad z_o = d. $$ The optical axes are $z$-forward, $x$-right, and $y$-down. ROS body axes are $x$-forward, $y$-left, and $z$-up, giving the fixed optical-to-body conversion $$ \begin{bmatrix} x_b\\ y_b\\ z_b \end{bmatrix} = \begin{bmatrix} 0 & 0 & 1\\ -1 & 0 & 0\\ 0 & -1 & 0 \end{bmatrix} \begin{bmatrix} x_o\\ y_o\\ z_o \end{bmatrix}. $$ The camera transform from the URDF has translation $$ t_{bc} = \begin{bmatrix} 0.064\\ -0.065\\ 0.094 \end{bmatrix} \text{ meters} $$ and no rotation. Compose the full transform in this order: $$ p_{\text{map}} = T_{mb} \, T_{bc} \, R_{co} \, p_{\text{optical}} $$ Here $R_{co}$ is the optical-to-body rotation, $T_{bc}$ the fixed `base_link` to camera transform, and $T_{mb}$ the map to `base_link` transform. Implement the chain with homogeneous coordinates and construct $T_{mb}$ from the synchronized robot pose. Check the odometry message frame before treating that pose as a map pose. When it is expressed in an odometry frame, apply the corresponding map-to-odometry transform first and document how you obtained it. An incorrect optical-frame conversion may produce a map that appears plausible but is mirrored or rotated by 90 degrees. Test the complete transform chain against a hand-computed result. The test must pass a known robot pose, bounding-box pixel, and depth through the implementation and compare the resulting map coordinate with the expected value using a stated tolerance. Select a pixel away from the image center so that the test exercises every axis conversion. ## Task 3: Semantic object database Create the pgvector extension and use these two tables: ```sql theme={null} CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE semantic_objects ( object_id BIGSERIAL PRIMARY KEY, class_name TEXT NOT NULL, map_x DOUBLE PRECISION NOT NULL, map_y DOUBLE PRECISION NOT NULL, map_z DOUBLE PRECISION NOT NULL, observation_count INTEGER NOT NULL CHECK (observation_count > 0), mean_embedding vector(512) NOT NULL, first_seen_ns BIGINT NOT NULL, last_seen_ns BIGINT NOT NULL ); CREATE INDEX semantic_objects_class_name_idx ON semantic_objects (class_name); CREATE TABLE object_observations ( observation_id BIGSERIAL PRIMARY KEY, object_id BIGINT NOT NULL REFERENCES semantic_objects (object_id) ON DELETE CASCADE, seen_at_ns BIGINT NOT NULL, map_x DOUBLE PRECISION NOT NULL, map_y DOUBLE PRECISION NOT NULL, map_yaw DOUBLE PRECISION NOT NULL, camera_x DOUBLE PRECISION NOT NULL, camera_y DOUBLE PRECISION NOT NULL, camera_z DOUBLE PRECISION NOT NULL, camera_qx DOUBLE PRECISION NOT NULL, camera_qy DOUBLE PRECISION NOT NULL, camera_qz DOUBLE PRECISION NOT NULL, camera_qw DOUBLE PRECISION NOT NULL, bearing_rad DOUBLE PRECISION NOT NULL, range_m DOUBLE PRECISION NOT NULL, bbox_xmin INTEGER NOT NULL, bbox_ymin INTEGER NOT NULL, bbox_xmax INTEGER NOT NULL, bbox_ymax INTEGER NOT NULL, confidence REAL NOT NULL, embedding vector(512) NOT NULL ); CREATE INDEX object_observations_object_id_idx ON object_observations (object_id); ``` Use ROS time in nanoseconds for all timestamp columns, and express the stored robot pose in the map frame. Derive the camera position and orientation from the localization transform chain. The report must state whether `range_m` denotes horizontal or three-dimensional distance and must define the zero direction and positive sign convention for `bearing_rad`. Each row in `semantic_objects` represents the current estimate of one physical object. The corresponding rows in `object_observations` preserve the measurements used to form that estimate. This separation supports indexed retrieval of every pose from which an object was observed without requiring a scan of the raw detector output. The same database supplies the [semantic graph layer](/aiml-common/lectures/simulation/semantic-graphs), in which objects and spatial relations are represented as graph entities. ## Task 4: Deduplication Associate each new detection only with stored objects that have the same `class_name` and whose fused map positions lie within a selected spatial radius. If the candidate set is empty, create a semantic object and its first observation. If it contains one object, merge the detection with that object. If it contains multiple objects, select the candidate with the highest CLIP cosine similarity. Insert a new observation row for every accepted detection. Select the association radius using measurements of depth noise, transform error, bounding-box variation, and viewpoint change. Document both the selected value and the measurements used to justify it. On a merge, update the fused position with a running average: $$ \bar{p}_{n+1} = \frac{n\bar{p}_n + p_{n+1}}{n+1}. $$ Update the mean embedding by the same rule, L2-normalize it again, increment `observation_count`, update `last_seen_ns`, and insert the new evidence row. In pgvector, `<=>` is cosine distance. For normalized vectors, it equals one minus their dot product, so the smallest distance identifies the most similar candidate. A group of four dining chairs illustrates the limitation of spatial-radius association. Although the chairs are distinct objects, several may fall within a radius large enough to accommodate the position error obtained when one chair is viewed from opposite sides. A smaller radius tends to produce duplicate records, whereas a larger radius tends to produce false merges. Measure how the selected radius and the embedding-based tie-break rule affect both outcomes. ## Task 5: Querying the database Implement and demonstrate the following parameterized queries. Do not interpolate values into SQL. Find the nearest instance of a requested class to a map point: ```sql theme={null} SELECT object_id, class_name, map_x, map_y, map_z, sqrt( power(map_x - $2, 2) + power(map_y - $3, 2) ) AS distance FROM semantic_objects WHERE class_name = $1 ORDER BY distance LIMIT 1; ``` List every object inside a rectangular map region: ```sql theme={null} SELECT object_id, class_name, map_x, map_y, map_z, observation_count FROM semantic_objects WHERE map_x BETWEEN $1 AND $2 AND map_y BETWEEN $3 AND $4 ORDER BY class_name, object_id; ``` Produce a histogram of stored objects per class: ```sql theme={null} SELECT class_name, count(*) AS object_count FROM semantic_objects GROUP BY class_name ORDER BY object_count DESC, class_name; ``` List every robot pose from which one object was seen: ```sql theme={null} SELECT observation_id, seen_at_ns, map_x, map_y, map_yaw, camera_x, camera_y, camera_z, bearing_rad, range_m, confidence FROM object_observations WHERE object_id = $1 ORDER BY seen_at_ns; ``` ## Evaluation Use the world SDF to establish ground-truth instance counts for every detectable class. Count distinct physical instances separately, including instances that use similar mesh assets or model names. Define instance precision as the number of correctly matched stored objects divided by the total number of stored objects. Define instance recall as the number of correctly matched stored objects divided by the number of ground-truth objects. The duplicate rate is the number of stored objects divided by the number of ground-truth objects. Its ideal value is `1.0`; values above `1.0` indicate duplicate records, while values below `1.0` may indicate missed objects or false merges. The detection rate is the fraction of relevant keyframes that contain at least one correct detection of the class, where a relevant keyframe is one in which an instance of that class is visible. Also report false merges, in which one stored record contains observations from multiple physical objects, and misses, in which a ground-truth object has no stored record. Interpret these measures using the observation histories and estimated map positions. The chair row is an example. Add one row for each remaining class. | Class | True instances | Stored objects | Correct matches | Precision | Recall | Duplicate rate | Detection rate | False merges | | ----- | -------------: | -------------: | --------------: | --------: | -----: | -------------: | -------------: | -----------: | | chair | 8 | 9 | 7 | 0.778 | 0.875 | 1.125 | 0.840 | 1 | | | | | | | | | | | Include a map plot of the fused object positions, with each point labeled by class and object identifier. Overlay the traversal route or the accepted robot poses to distinguish incomplete spatial coverage from detector failures. Discuss classes that are absent from the environment, rarely visible, or consistently missed by the detector. ## Deliverables Submit: * The detector source code, including Zenoh subscriptions, CDR message definitions, keyframe gating, YOLOv8 inference, and batched CLIP embedding. * The depth synchronization and map back-projection implementation. * The transform-chain unit test and its hand-computed expected result. * The SQL schema and database-ingest code. * The deduplication implementation with its documented radius and similarity rule. * Executable versions of the four required database queries. * A database export containing semantic objects and their observations. * A map plot showing object estimates, identifiers, and coverage. * A report containing the route, per-class evaluation, duplicate-rate analysis, failure cases, and design choices. * A short README with commands needed to run your implementation against the compose services. ## Grading | Component | Points | | ------------------------------------------ | -----: | | Detector node | 20 | | Back-projection with unit test | 25 | | Schema and ingest | 20 | | Deduplication with measured duplicate rate | 25 | | Report | 10 | | Total | 100 | *** [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/assignments/topics/ROS/object-detection/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).