3D reconstruction
We scanned the workspace with an iPhone mounted on a gimbal and fused the captured images into a Gaussian Splatting reconstruction. The result is a continuous volumetric model of the table, the objects resting on it, and the shelves the arm places into. Where a photograph gives appearance, the reconstruction gives occupancy: whether a mug fits on the middle shelf, and how far the gripper can travel before it contacts something already there. The scan is performed before operation and is not updated while the arm runs.
Flying through the reconstructed workspace. Generated by Ayush Ramashankar (NJIT).
From speech to action
The instruction becomes a goal, not a path. “The middle shelf” names a target region, and nothing in the utterance constrains where the elbow should be. Recovering the path is the policy’s problem. The policy takes two inputs, the visual state of the workspace from a set of fixed cameras and the language embedding of the goal, and emits joint angles and gripper commands. A single model spans perception through control, so the visual state conditions the entire action sequence rather than being reduced to an object list that later stages consume. Placement falls out of the same computation. Where on the shelf the mug goes depends on the clearance around candidate slots and on any preference the operator expressed, and both reach the policy as inputs rather than as a filter applied to its output. Execution proceeds in three phases that are visible in the workspace: the grasp at the table, the lift clear of surrounding objects, and the transport and release into the shelf.
The arm running the three phases end to end.
Training the policy
The policy was trained offline on large-scale robot demonstrations collected by human teleoperation. Each demonstration pairs the camera view and the instruction given to the operator with the actions the operator took, and training aligns those streams so the model associates an instruction and a scene with the motion they call for. Nothing is trained on the arm during a session. The deployed policy is fixed, so its behavior in a scene it has not encountered comes from that alignment generalizing rather than from per-task programming.Scope of the demonstration
The recordings on this page are qualitative. They show the pipeline completing the task in the configurations filmed, and we have not measured success rates across object placements, shelf occupancies, or phrasings of the instruction, so they say nothing about how often it succeeds in general. Because the reconstruction is captured before operation, rearranging the scene afterwards leaves the model stale even though the fixed cameras still observe the change. Behavior under clutter and on objects absent from the demonstration data is uncharacterized.Technical stack
The arm’s kinematics, the mapping between joint angles and gripper pose that the policy’s commands presuppose, are worked through in AR4 robot arm kinematics and the URDF lab.
Contact us to adapt this architecture for your robots.