Sim-to-Real Transfer

Policies trained in simulation often fail on real hardware. Differences in visual appearance, physics, sensor noise, and dynamics create a domain gap that reduces performance during deployment.

​

The domain gap

A policy trained in Gazebo sees flat textures, perfect lighting, and noiseless depth. A real kitchen has specular reflections, clutter, and a depth camera that hallucinates on glass surfaces. The domain gap has three main forms:

• Visual mismatch: Synthetic renders differ from real camera images.
• Dynamics mismatch: Simulated friction, mass, and contact differ from those of the real robot.
• Sensor mismatch: Simulated sensors are perfect, while real IMUs, cameras, and lidars are noisy.

Each technique below addresses one or more of these gaps.

​

Domain randomization

Domain randomization varies simulation parameters such as textures, lighting, object positions, camera noise, and friction coefficients. The range is chosen so that the real world falls within the training distribution. The policy then learns to ignore visual and physical variation instead of memorizing one simulated environment. Domain randomization is simple to implement but expensive. You need enough variation to cover reality, and you cannot know in advance whether the range is sufficient. It also tends to produce conservative policies that sacrifice precision to handle variation.

​

Domain adaptation

Domain adaptation aligns the feature distributions of simulated and real data so they become indistinguishable to the policy. Techniques include:

• Adversarial adaptation: Train a discriminator to distinguish simulated features from real features. Train a feature extractor to fool the discriminator.
• Style transfer: Process simulated images with a neural style transfer network trained on real images.
• Feature matching: Minimize the distributional distance (MMD, CORAL) between simulated and real feature spaces.

Domain adaptation requires some real-world data for alignment, but far less than training a policy from scratch on real data.

​

System identification

System identification calibrates the simulator’s physics parameters to match the real robot. You measure friction, damping, motor response curves, and sensor noise profiles on the real system. You then set the simulator parameters to match these measurements. This method reduces the dynamics gap directly instead of training the policy to tolerate it. System identification is most effective for dynamics-dominated tasks (locomotion, contact-rich manipulation) where visual appearance matters less than physical accuracy.

​

3D Gaussian splatting for photorealistic world generation

The techniques above start with a hand-authored simulation and compensate for its limited visual detail. Another approach starts with a 3D scan of the real environment and trains the policy in a photorealistic reconstruction. 3D Gaussian Splatting (3DGS) reconstructs a dense, renderable 3D scene from a set of posed images. Each Gaussian carries position, shape, color, and opacity. The renderer splats the Gaussians onto an image plane without ray marching. This method supports real-time (100+ FPS) novel view synthesis. This method produces a different sim-to-real pipeline: 3DGS has three roles in sim-to-real transfer: Photorealistic world generation. You scan the target environment (warehouse, kitchen, hospital corridor) with a camera, train a Gaussian splat, and render training views from it. This process replaces hand-authored simulator worlds with approximate textures and lighting. The training images are novel views derived from the real environment. This reduces the visual domain gap. Infinite viewpoint augmentation. A single scan produces a continuous 3D field that can be rendered from any pose. The robot can train on views of the real space from poses it has never physically visited. Unlike image augmentation (crop, color jitter), these novel views are geometrically consistent and have correct occlusion and parallax. Semantic simulation environments. Language-embedded splats (LEGaussians, LangSplat) make the reconstructed world queryable. For example, the query “where is the couch?” returns a 3D location. A VLA agent can train on navigation and manipulation tasks in a photorealistic, semantically labeled environment derived from a real scan. This process does not require manual annotation. Representations have progressed from classical maps to 3DGS-based simulation: The Gaussian Splatting for Robot Navigation project covers this pipeline. It includes capturing a real room, training a splat, and comparing simulation with the real environment.

​

Combining approaches

These techniques can be combined:

• Use 3DGS to reduce the visual gap at the source.
• Apply domain randomization to factors that the scan does not capture (lighting changes, object rearrangement, sensor noise).
• Use system identification for dynamics-critical tasks.
• Apply domain adaptation as a final fine-tuning step with a small amount of real-world data.

Layer the techniques so that each remaining gap is addressed by a suitable method. Key references: (Bousmalis et al., 2017; Marco et al., 2017; Weber et al., 2017; Pan et al., 2017; Hester et al., 2017)

​

References

• Bousmalis, K., Irpan, A., Wohlhart, P., Bai, Y., Kelcey, M., et al. (2017). Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping.
• Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., et al. (2017). Deep Q-learning from Demonstrations.
• Marco, A., Berkenkamp, F., Hennig, P., Schoellig, A., Krause, A., et al. (2017). Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization.
• Pan, X., You, Y., Wang, Z., Lu, C. (2017). Virtual to Real Reinforcement Learning for Autonomous Driving.
• Weber, T., Racanière, S., Reichert, D., Buesing, L., Guez, A., et al. (2017). Imagination-Augmented Agents for Deep Reinforcement Learning.

---