Skip to main content
  • World + object placement — 20
  • Detection node + metadata correctness — 25
  • Zenoh publishing + key design — 20
  • Ingest worker + idempotent DB writes — 25
  • Reproducibility report — 10
You will convert your detection event database into a semantic spatial graph and implement semantic re-localization using visual similarity and graph structure. Zenoh remains your live transport layer. PostgreSQL becomes your semantic map store.

​
Background Videos — Semantic + ROS2 Mapping Concepts

​
Zenoh Advanced Features Workshop

​
ROS 2 Mapping and Frame Semantics (map vs odom)

Consult REP 105 — Coordinate Frames for Mobile Platforms and the Nav2 Transforms Setup Guide for the relationship between the map, odom, and base_link frames used in this assignment.

​
1. Global Pose Requirement

Produce map-frame poses for keyframes using one:
  • SLAM toolbox mapping mode, or
  • Localization against a known map
For each keyframe store:
  • map_x
  • map_y
  • map_yaw
  • timestamp
This is required for global semantic mapping.

​
2. Vector Embeddings (pgvector)

You will:
  • Crop each detection bounding box
  • Compute an embedding (CLIP or similar)
  • Store vectors in pgvector
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE detection_embeddings (
  det_pk bigint PRIMARY KEY REFERENCES detections(det_pk),
  model text,
  embedding vector(512)
);
Document your embedding model and dimension.

​
3. Semantic Graph with Apache AGE

Enable AGE per session: 
LOAD 'age';
SET search_path = ag_catalog, "$user", public;
CREATE EXTENSION age;

​
Node types

  • Run
  • Keyframe
  • Pose (map frame)
  • Place
  • Object
  • Observation

​
Edge types

  • Run → Keyframe
  • Keyframe → Pose
  • Keyframe → Observation
  • Observation → Object
  • Object → Place
  • Place → Place (adjacent)

​
4. Place Construction

Cluster map-frame poses into places using:
  • grid binning, or
  • DBSCAN
Each keyframe must belong to exactly one place.

​
5. Object Landmark Fusion

Merge observations into object landmarks using:
  • spatial distance threshold
  • embedding similarity threshold
Each landmark must maintain:
  • class
  • mean position
  • first_seen / last_seen
  • observation count

​
6. Semantic Re-Localization Task

​
Scenario

Run A:
  • Explore maze
  • Build semantic graph
Run B:
  • Restart at unknown pose
  • Capture 1–3 object crops
  • Compute embeddings
  • Run vector KNN search
  • Infer top-3 likely places
  • Output best pose hypothesis

​
Minimal algorithm

  • Compute embeddings for query crops
  • KNN search in pgvector
  • Join to Object → Place
  • Rank places by similarity score

​
Required Queries

Provide working scripts for:

​
Vector

Top-k visually similar detections for a query crop.

​
Graph

Reachable places within N hops containing an object class.

​
Re-localization

Top-3 candidate places from query crops.

​
Deliverables

  • pgvector + AGE schema
  • Embedding generator
  • Graph builder
  • Semantic relocalizer
  • Demo report with success and failure analysis
Edit this page on GitHub or file an issue.