> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# From a Phone Video to a Simulated Object

> Capture one object with your phone, rebuild it as a sim-ready USD asset with open-source models, and measure how far the twin is from the real thing.

A robot learns much of what it knows about objects in simulation, so the objects in the simulator have to look and behave like the real ones. In an NVIDIA Omniverse livestream on closing the real-to-sim-to-real loop (Omniverse, n.d.), the company AllSides shows how it builds such objects ([the segment starts at 43:52](https://www.youtube.com/watch?v=q1UFopOrrQM\&t=2632s)). It photographs a real object about 15,000 times under controlled lights and turns the photos into a relightable OpenUSD asset with sub-millimeter geometry, physically based materials, a measured mass and collision shapes. The asset then passes a review in simulation. AllSides names transparent and deformable objects as the cases it cannot handle.

In this project you build a smaller version of that pipeline for one object of your own. You capture it with a phone video next to a printed board, and six stages turn the video into a USD asset that loads, falls and slides in the Newton simulator. A seventh, stage 4b, sits between stages 4 and 5 and examines the materials without changing the asset. Each stage rests on a published method and a public repository, and every stage is meant to run on a free Colab T4 GPU. All seven stages were timed on a T4, and each stage page states its time. A phone video under one ordinary lamp cannot match a studio capture. What you deliver is an understanding of each method, good enough to explain what it assumes and where it fails, and a measurement of how far your twin is from your object.

| Stage | What it does | What it produces |
| - | - | - |
| 1. Capture and poses | Recovers a camera pose for every frame and fixes the scale from the board | A sparse model in meters |
| 2. Masks | Separates the object from the table and the board in every frame | One mask per frame |
| 3. Geometry | Trains two Gaussian splats and fuses rendered depth into a closed surface | A splat and a watertight mesh |
| 4. Materials | Predicts albedo, roughness and metallic in many photos and bakes them onto the mesh | A textured mesh with three material maps |
| 4b. Inverse rendering | Fits the three maps and a model of the room's light to your photos by differentiable rendering | Refined maps, the learned light, and a record of how albedo and light trade against each other |
| 5. Physics | Splits the mesh into convex pieces and adds your mass, friction and balance measurements | Colliders, mass, center of mass, inertia and a friction coefficient |
| 6. Export and QA | Writes the USD asset, tests it in the simulator and against held-out photos | The asset and two scorecards |

Every stage logs what it reads and writes to one Rerun recording in one world frame, so you can replay the whole pipeline, from the frames to the simulated ramp, in a single viewer.

Stage 4b fits the maps better to your photos, but it does not find out what the object is made of. Under one light, a brighter albedo and a dimmer light give the same diffuse color, and the photos say little about metallic. Stage 4b measures both effects on your object. Stages 5 and 6 keep the stage 4 maps for the physics and the exported asset, and stage 6 scores the refined maps beside them.

Stage 6 ends with a comparison against a model that skips the measurement. The course ran TRELLIS.2, which generates a textured 3D object from a single photo, on one frame of each of two reference objects. Stage 6 scores each generated object against the measured twin of the same object, with the Chamfer distance to the dataset's scan and the same held-out photo scores. The generator needs a larger GPU than Colab offers, so you read its output instead of running it. The second reference object, a hammer, is a robustness case: its measured pipeline does not pass every check that was calibrated on the carton.

## Grading

The project is worth 100 points. For stages 1 to 4 and stage 4b, half of a stage's points are for its output and half for its answers to the stage's questions. Stages 5 and 6 are graded on their output and answers together.

| Component | Output | Answers | Points |
| - | - | - | - |
| Stage 1. Capture and poses | 7.5 | 7.5 | 15 |
| Stage 2. Masks | 5 | 5 | 10 |
| Stage 3. Geometry | 10 | 10 | 20 |
| Stage 4. Materials | 5 | 5 | 10 |
| Stage 4b. Inverse rendering | 2.5 | 2.5 | 5 |
| Stage 5. Physics | | | 15 |
| Stage 6. Export and QA, with the two scorecards | | | 15 |
| Shortfalls section | | | 10 |
| Total | | | 100 |

An output earns its points when the stage's check passed, on your own capture or, after a fallback at an earlier stage, on the reference intermediate, and the files it wrote are in your project folder and in the recording. An answer is graded on how well it explains the technique. Each question also asks you to point at the place in your own run where the idea shows up, and an answer without that pointer earns at most partial credit, however good the rest of it is.

## Background reading

Read the method behind each stage before you start it. Each stage page also explains the mechanism and the math it relies on, so you can run a stage before you have finished its papers and come back to them for depth.

### Stage 1: capture and poses

Schönberger and Frahm, [Structure-from-Motion Revisited](https://doi.org/10.1109/CVPR.2016.445) (CVPR 2016), is the method inside [COLMAP](https://github.com/colmap/colmap), which you run through its Python bindings, [pycolmap](https://pypi.org/project/pycolmap/). Hartley and Zisserman's book covers the projection model and triangulation (Hartley & Zisserman, 2004). The board is a ChArUco pattern: read the [OpenCV ChArUco tutorial](https://docs.opencv.org/4.x/df/d4a/tutorial_charuco_detection.html), and for the markers themselves Garrido-Jurado et al., [Automatic generation and detection of highly reliable fiducial markers under occlusion](https://doi.org/10.1016/j.patcog.2014.01.005) (Pattern Recognition 2014). The fit that puts the model in meters is Umeyama's [least-squares similarity](https://doi.org/10.1109/34.88573) (IEEE TPAMI 1991).

### Stage 2: masks

Ravi et al., [SAM 2: Segment Anything in Images and Videos](https://arxiv.org/abs/2408.00714) (2024), with code and weights in [facebookresearch/sam2](https://github.com/facebookresearch/sam2). SAM 2 extends the image segmenter of Segment Anything (Kirillov et al., 2023) with a memory of past frames.

### Stage 3: geometry

3D Gaussian splatting (Kerbl et al., 2023) gives the appearance splat. Huang et al., [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://doi.org/10.1145/3641519.3657428) (SIGGRAPH 2024), replaces the blobs with flat disks whose depth lies on a surface. Both are trained with gsplat (Ye et al., 2024), in [nerfstudio-project/gsplat](https://github.com/nerfstudio-project/gsplat). The depth maps are fused with the volumetric method of Curless and Levoy, [A volumetric method for building complex models from range images](https://doi.org/10.1145/237170.237269) (SIGGRAPH 1996), as implemented in [Open3D](https://arxiv.org/abs/1801.09847).

### Stage 4: materials

Ke et al., [Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis](https://arxiv.org/abs/2505.09358) (2025), adds the intrinsic image models to the original [Marigold depth estimator](https://doi.org/10.1109/CVPR52733.2024.00907) (CVPR 2024). The code is in [prs-eth/Marigold](https://github.com/prs-eth/Marigold). The model was fine-tuned on the synthetic scenes of [InteriorVerse](https://doi.org/10.1145/3550469.3555407) (SIGGRAPH Asia 2022), which is why its guesses look like indoor renderings. The texture charts come from [xatlas](https://github.com/jpcy/xatlas).

### Stage 4b: inverse rendering

Karis, [Real Shading in Unreal Engine 4](https://blog.selfshadow.com/publications/s2013-shading-course/karis/s2013_pbs_epic_notes_v2.pdf) (SIGGRAPH 2013 course notes), describes the GGX shading and the split-sum approximation that the stage uses, and his [Physically Based Shading on Mobile](https://www.unrealengine.com/en-US/blog/physically-based-shading-on-mobile) (2014) gives the polynomial fit it uses for one of the two factors. The mesh is rendered with gradients by nvdiffrast, from Laine et al., [Modular Primitives for High-Performance Differentiable Rendering](https://doi.org/10.1145/3414685.3417861) (SIGGRAPH Asia 2020), with code in [NVlabs/nvdiffrast](https://github.com/NVlabs/nvdiffrast). Munkberg et al., [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://arxiv.org/abs/2111.12503) (CVPR 2022), fits a mesh, its materials and an environment light to photos with the same kind of shading, and the stage's shading follows its approach but uses none of its code.

### Stage 5: physics

Wei et al., [Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search](https://doi.org/10.1145/3528223.3530103) (SIGGRAPH 2022), is the method in [CoACD](https://github.com/SarahWeiii/CoACD). Volumes, centers of mass and inertia tensors come from [trimesh](https://github.com/mikedh/trimesh).

### Stage 6: export and QA

The asset follows the [OpenUSD](https://openusd.org/release/index.html) format and its [physics schema](https://openusd.org/release/api/usd_physics_page_front.html). The simulator is [Newton](https://github.com/newton-physics/newton), built on [NVIDIA Warp](https://github.com/NVIDIA/warp). The appearance score uses LPIPS from Zhang et al., [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://doi.org/10.1109/CVPR.2018.00068) (CVPR 2018), with code in [richzhang/PerceptualSimilarity](https://github.com/richzhang/PerceptualSimilarity). Every stage logs to [Rerun](https://github.com/rerun-io/rerun).

The generated baseline is TRELLIS.2, from Xiang et al., [Native and Compact Structured Latents for 3D Generation](https://arxiv.org/abs/2512.14692) (2025), with code and weights in [microsoft/TRELLIS.2](https://github.com/microsoft/TRELLIS.2) under the MIT license. It encodes the photo with [DINOv3](https://arxiv.org/abs/2508.10104) (Siméoni et al., 2025), whose weights carry the DINOv3 License, and it was trained on [Objaverse-XL](https://arxiv.org/abs/2307.05663) (Deitke et al., 2023).

### Optional paths

When COLMAP fails on an object with little texture, VGGT (Wang et al., 2025) or [MapAnything](https://arxiv.org/abs/2509.13414) can recover the poses instead. 3DGUT (Wu et al., 2024) trains a splat that exports to USD directly, and [nvdiffrec](https://arxiv.org/abs/2111.12503) recovers mesh and materials together. None of these is required. If you use one, name it and answer the same questions.

## How the tasks work

Each stage is its own Colab notebook, and the stages pass their results to each other through a project folder on your Google Drive, `My Drive/real-to-sim-twin/<object>/`. Set the folder name once in the first cell of each stage. Open each stage from its card below and save your own copy before you write anything (File, then Save a copy in Drive): edits to a notebook opened from GitHub are not kept. A stage reads the previous stage's outputs from that folder and writes its own there, so Colab can disconnect between stages without losing work, and you can rerun any stage alone.

Most tasks ask you to write one function. A check cell at the end of the stage tests the results and prints what is wrong when something is. The stage writes its outputs only when the check passes. Some results depend on your object and your capture. When one of them looks unusual, the check prints a warning and lets you continue, and the questions ask you to explain it.

Until your own capture is in place, every stage runs on a reference object, a boxed beverage carton from [OmniObject3D](https://github.com/omniobject3d/OmniObject3D) (CC BY 4.0), so you can read and run a stage before you have filmed anything. The stage prints a line when it falls back to the reference. Such a run leaves the reference files in `capture/`, among them a hidden `.reference` file and the carton's `physical.json`. When you have filmed, put your capture in a fresh project folder (change `OBJECT` in the first cell), or delete the reference files from `capture/` before you add yours. Stage 1 stops with this advice if it finds both.

Each stage ends with two to four questions about the technique: how the method works, the math behind it, what it assumes and how it fails. Write your answers in the empty cells below each question. Every question ends by asking you to point at one place in your own run, a printed number, a frame or a region in the viewer, where the idea shows up.

## What you need

* The ChArUco board, printed at 100% scale on [US Letter](https://artifacts.aegeanai.com/projects/real-to-sim-twin/board/charuco-letter.pdf) or [A4](https://artifacts.aegeanai.com/projects/real-to-sim-twin/board/charuco-a4.pdf) paper. The board is the same on both. The bar under the board must measure 50 mm.
* A ruler, to measure one printed square and one marker, and three dimensions of your object.
* A kitchen scale, for the object's mass.
* A hardback book or a stiff board to use as a ramp.
* A phone app that reads tilt in degrees. On an iPhone, use the Level tab of the built-in Measure app. On Android, install [Bubble Level](https://play.google.com/store/apps/details?id=net.androgames.level).
* A pencil or the edge of a table, to find where the object balances.
* A desk lamp that you can move, for the photo under a second light.
* A phone that records video with exposure and focus locked.
* A Google account for Colab and Google Drive, with a T4 GPU runtime.

Stage 1 explains how to film the orbit. Stage 5 explains the measurements that go into `capture/physical.json`: the mass, five tilt angles at which the object starts to slide, three dimensions, three balance readings from a named end of the object, and, if your object is made of a metal part and a non-metal part, the two materials. Stage 6 explains the photo under the second lamp.

## Choosing your object

Pick one object that is:

* rigid and opaque, so no glass, cloth or plush toys;
* not very shiny, since a mirror-like surface defeats both the pose solver and the material model;
* small enough to fit within a Letter sheet next to the board;
* light enough for a kitchen scale and heavy enough not to blow over on the ramp;
* textured, with print, grain or marks on its surface. If it is plain, add a few stickers or pencil marks.

AllSides excludes transparent and deformable objects too.

## Weekly checkpoints

The project runs for five weeks. Use these checkpoints to pace your work.

| Week | Stages | By the end of the week |
| - | - | - |
| 1 | 1 and 2 | A metric sparse model, the masks, the quick splats of stage 2 as an early check of your capture, and the stage 1 and 2 answers |
| 2 | 3 | The splat and the mesh, with the held-out frames kept aside, and the stage 3 answers |
| 3 | 4 and 4b | The textured mesh, the refined maps and the learned light, the photo under the second lamp, and the stage 4 and 4b answers |
| 4 | 5, and the export in 6 | The measurements, the colliders and mass properties, the asset loading and sliding in the simulator, and the stage 5 answers |
| 5 | 6 | Both scorecards, the stage 6 answers and the shortfalls section |

Film your object in the first days of week 1. If stage 2's quick splat looks wrong, film again right away, while a reshoot costs you only an afternoon.

## Stage notebooks

> **[Stage 1. Capture and poses](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/01-capture-and-poses/01-capture-and-poses.ipynb)**
>
> Camera poses and a sparse model in meters.

> **[Stage 2. Masks](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/02-masks/02-masks.ipynb)**
>
> Object masks in every frame with SAM 2.

> **[Stage 3. Geometry](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/03-geometry/03-geometry.ipynb)**
>
> Two splats and a watertight mesh.

> **[Stage 4. Materials](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/04-materials/04-materials.ipynb)**
>
> Albedo, roughness and metallic maps on the mesh.

> **[Stage 4b. Inverse rendering](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/04b-inverse-rendering/04b-inverse-rendering.ipynb)**
>
> Refined maps and a learned light, fitted to your photos.

> **[Stage 5. Physics](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/05-physics/05-physics.ipynb)**
>
> Colliders, mass properties and friction.

> **[Stage 6. Export and QA](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/projects/cv/real-to-sim-twin/06-export-and-qa/06-export-and-qa.ipynb)**
>
> The USD asset, the simulated tests and the scorecards.

## What you hand in

Your submission has two parts, because some of the files are too large for a repository.

In a `projects/real-to-sim-twin` directory of your class repository, commit:

* the seven stage notebooks (stages 1 to 6 and 4b), executed on your own object, with every cell output saved and your answers in the answer cells;
* the two scorecards from `stage6/`, `scorecard_appearance.json` and `scorecard_behavior.json`;
* a `submission.md` that names your object, links your Drive project folder, lists any stage where you fell back to the reference with the reason, and holds your shortfalls section.

Keep the large files in your Drive project folder: the merged recording `stage6/pipeline.rrd`, the USD asset with its textures in `stage6/asset/`, the capture video and frames, the held-out frames, and the photo under the second lamp. Share the folder view-only with the course staff and paste its link in `submission.md`. The graders read the folder's modification times, so do not change anything in it after the deadline. Keep the recording under 500 MB, so that it opens in a browser, and the asset with its textures under 200 MB. Stage 6 checks both limits.

The shortfalls section is the last part of your write-up. Name the two or three places where your twin differs most from your object, give the number that shows each one, and name the stage it comes from. Stage 6's last question walks you through it. If you fell back at any of stages 1 to 4, your stage 5 and 6 numbers describe the reference object, so also give your own measurements (mass, tilt angles, dimensions and balance readings) beside the reference ones.

Submit the full GitHub URL of the directory to Canvas.

## When a stage fails on your capture

If one of stages 1 to 4 fails on your capture and you cannot fix it by filming again, you may continue from the reference object. When a stage's check fails, it writes nothing, and the next stage finds its input missing and falls back to the matching reference intermediate. Your own outputs stay in the project folder, renamed with a `.yours` suffix.

A fallback costs you the output half of that stage's points. You keep the answers half: answer the stage's questions with your own failed run and explain why it failed, and that explanation is graded as one of the answers. The later stages then run on the reference object and are graded normally, on the reference.

You may fall back on at most two stages. With three or more, the object is no longer yours, and the project is capped at 50 points.

Stage 4b follows its own rule, because it has no capture of its own. When stage 4 fell back, stage 4b runs on the reference stage 4 and is graded on that run. When stage 4b itself fails on your object, you continue without it. Stage 5 does not read its output, and stage 6 leaves out the refined rows and prints why. The failure costs the output half of stage 4b's points, and you answer its questions with your failed run and explain the failure, as after any fallback. It does not count toward the limit of two fallbacks. Skipping stage 4b is not a fallback: a stage 4b you never ran earns none of its points.

Stages 5 and 6 have no fallback of their own, because they need your real object on the scale, on the ramp and under the second lamp. After a fallback at any of stages 1 to 4, they run entirely on the reference object, including its measurements, because your measurements do not fit the reference geometry. Your own `capture/` folder is kept as `capture.yours/`. Report your measurements beside the reference ones in the shortfalls section.

## References

* Hartley, R., Zisserman, A. (2004). *Multiple View Geometry in Computer Vision*.
* Kerbl, B., Kopanas, G., Leimkuehler, T., Drettakis, G. (2023). *3D Gaussian splatting for real-time radiance Field rendering*. ACM Trans. Graph.
* Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., et al. (2023). *Segment Anything*. arXiv \[cs.CV].
* Omniverse, N. (n.d.). *Closing the real-to-Sim-to-real loop for physical AI*.
* Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., et al. (2025). *VGGT: Visual geometry grounded transformer*. arXiv \[cs.CV].
* Wu, Q., Esturo, J., Mirzaei, A., Moenne-Loccoz, N., Gojcic, Z. (2024). *3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting*.
* Ye, V., Li, R., Kerr, J., Turkulainen, M., Yi, B., et al. (2024). *gsplat: An Open-Source Library for Gaussian Splatting*.

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/projects/cv/real-to-sim-twin).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.