Skip to main content

​
1. Problem and premise

Robots expose their behavior through low-level interfaces: topics, services, launch files, parameters. Engineers who have memorized these can operate a robot quickly. Everyone else — field operators, domain experts, students, and even the engineers themselves when they are debugging — would rather say what they want in plain language and let a capable agent figure out the plumbing. The premise is simple: a modern LLM-based agent with tools for introspecting and driving the robot can translate intent into correct behavior. The interesting question is not whether this works but which agent harness you use. A harness decides how the agent plans, what tools it has, how it reads the robot’s workspace, and how it recovers from failure. Different harnesses make different tradeoffs. Agent architecture The architecture is deliberately decoupled: the agent talks to a message bus, and a small adapter on the robot side carries out the commands. This keeps the harness interchangeable — you can swap one for another without touching the robot.

​
2. Harnesses

Three harnesses are on the bench. They are alternatives, not pieces of a single stack.

​
ROSA (initial experiment)

ROSA was our initial experiment in this space — a purpose-built robotics agent that ships with a hand-written tool set for robot introspection and control, with tools that validate their inputs and outputs before anything reaches the robot. ROSA architecture
  • Hand-crafted tools for the common robot operations
  • Works with any LLM backend — OpenAI, Anthropic, open models
  • Structured, validated responses
ROSA proved the premise, but we have since superseded it with Claude Code, which removes the need to maintain a bespoke tool set and lets the agent reach into the codebase directly.

​
Claude Code

Claude Code is Anthropic’s general-purpose coding agent. Instead of a hand-crafted robotics tool set, it arrives with generic tools — read files, run commands, edit files, search the workspace — and it treats the robot’s codebase the same way it treats any other project.
  • No wrapper code to write
  • Can read launch files, edit source, rebuild, and restart the stack
  • Can combine code changes and live robot commands in a single session
Claude-only backend. Pick Claude Code when you want the agent to also touch your codebase — for prototyping, debugging, or refactoring alongside live operation.

​
NeMoclaw

NeMoclaw is NVIDIA’s robotics agent harness, built on the NeMo agent stack. We are currently bringing it up against the same robot adapter and will present results soon. All harnesses run against the same message bus and the same robot adapter.

​
3. Demos

​
ROSA

Video by Oscar Poudel, Ph.D. candidate at NJIT.

​
Claude Code

Claude Code reading the workspace, reasoning about the robot’s interface, and driving it end-to-end.
Contact us to adapt this architecture for your robots.
Edit this page on GitHub or file an issue.