From a Phone Video to a Simulated Object

A robot learns much of what it knows about objects in simulation, so the objects in the simulator have to look and behave like the real ones. In an NVIDIA Omniverse livestream on closing the real-to-sim-to-real loop (Omniverse, n.d.), the company AllSides shows how it builds such objects (the segment starts at 43:52). It photographs a real object about 15,000 times under controlled lights and turns the photos into a relightable OpenUSD asset with sub-millimeter geometry, physically based materials, a measured mass and collision shapes. The asset then passes a review in simulation. AllSides names transparent and deformable objects as the cases it cannot handle. In this project you build a smaller version of that pipeline for one object of your own. You capture it with a phone video next to a printed board, and six stages turn the video into a USD asset that loads, falls and slides in the Newton simulator. Each stage rests on a published method and a public repository, and every stage is meant to run on a free Colab T4 GPU. Stages 1 and 2 were timed on a T4. Stages 3 to 6 were timed on a faster GPU and are expected to fit a T4, where they take longer. A phone video under one ordinary lamp cannot match a studio capture. What you deliver is an understanding of each method, good enough to explain what it assumes and where it fails, and a measurement of how far your twin is from your object. Every stage logs what it reads and writes to one Rerun recording in one world frame, so you can replay the whole pipeline, from the frames to the simulated ramp, in a single viewer.

​

Grading

The project is worth 100 points. For stages 1 to 4, half of a stage’s points are for its output and half for its answers to the stage’s questions. Stages 5 and 6 are graded on their output and answers together. An output earns its points when the stage’s check passed, on your own capture or, after a fallback at an earlier stage, on the reference intermediate, and the files it wrote are in your project folder and in the recording. An answer is graded on how well it explains the technique. Each question also asks you to point at the place in your own run where the idea shows up, and an answer without that pointer earns at most partial credit, however good the rest of it is.

​

Background reading

Read the method behind each stage before you start it. Each stage page also explains the mechanism and the math it relies on, so you can run a stage before you have finished its papers and come back to them for depth.

​

Stage 1: capture and poses

Schönberger and Frahm, Structure-from-Motion Revisited (CVPR 2016), is the method inside COLMAP, which you run through its Python bindings, pycolmap. Hartley and Zisserman’s book covers the projection model and triangulation (Hartley & Zisserman, 2004). The board is a ChArUco pattern: read the OpenCV ChArUco tutorial, and for the markers themselves Garrido-Jurado et al., Automatic generation and detection of highly reliable fiducial markers under occlusion (Pattern Recognition 2014). The fit that puts the model in meters is Umeyama’s least-squares similarity (IEEE TPAMI 1991).

​

Stage 2: masks

Ravi et al., SAM 2: Segment Anything in Images and Videos (2024), with code and weights in facebookresearch/sam2. SAM 2 extends the image segmenter of Segment Anything (Kirillov et al., 2023) with a memory of past frames.

​

Stage 3: geometry

3D Gaussian splatting (Kerbl et al., 2023) gives the appearance splat. Huang et al., 2D Gaussian Splatting for Geometrically Accurate Radiance Fields (SIGGRAPH 2024), replaces the blobs with flat disks whose depth lies on a surface. Both are trained with gsplat (Ye et al., 2024), in nerfstudio-project/gsplat. The depth maps are fused with the volumetric method of Curless and Levoy, A volumetric method for building complex models from range images (SIGGRAPH 1996), as implemented in Open3D.

​

Stage 4: materials

Ke et al., Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis (2025), adds the intrinsic image models to the original Marigold depth estimator (CVPR 2024). The code is in prs-eth/Marigold. The model was fine-tuned on the synthetic scenes of InteriorVerse (SIGGRAPH Asia 2022), which is why its guesses look like indoor renderings. The texture charts come from xatlas.

​

Stage 5: physics

Wei et al., Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Search (SIGGRAPH 2022), is the method in CoACD. Volumes, centers of mass and inertia tensors come from trimesh.

​

Stage 6: export and QA

The asset follows the OpenUSD format and its physics schema. The simulator is Newton, built on NVIDIA Warp. The appearance score uses LPIPS from Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (CVPR 2018), with code in richzhang/PerceptualSimilarity. Every stage logs to Rerun.

​

Optional paths

When COLMAP fails on an object with little texture, VGGT (Wang et al., 2025) or MapAnything can recover the poses instead. 3DGUT (Wu et al., 2024) trains a splat that exports to USD directly, and nvdiffrec recovers mesh and materials together. None of these is required. If you use one, name it and answer the same questions.

​

How the tasks work

Each stage is its own Colab notebook, and the stages pass their results to each other through a project folder on your Google Drive, My Drive/real-to-sim-twin/<object>/. Set the folder name once in the first cell of each stage. Open each stage from its card below and save your own copy before you write anything (File, then Save a copy in Drive): edits to a notebook opened from GitHub are not kept. A stage reads the previous stage’s outputs from that folder and writes its own there, so Colab can disconnect between stages without losing work, and you can rerun any stage alone. Most tasks ask you to write one function. A check cell at the end of the stage tests the results and prints what is wrong when something is. The stage writes its outputs only when the check passes. Some results depend on your object and your capture. When one of them looks unusual, the check prints a warning and lets you continue, and the questions ask you to explain it. Until your own capture is in place, every stage runs on a reference object, a boxed beverage carton from OmniObject3D (CC BY 4.0), so you can read and run a stage before you have filmed anything. The stage prints a line when it falls back to the reference. Such a run leaves the reference files in capture/, among them a hidden .reference file and the carton’s physical.json. When you have filmed, put your capture in a fresh project folder (change OBJECT in the first cell), or delete the reference files from capture/ before you add yours. Stage 1 stops with this advice if it finds both. Each stage ends with two or three questions about the technique: how the method works, the math behind it, what it assumes and how it fails. Write your answers in the empty cells below each question. Every question ends by asking you to point at one place in your own run, a printed number, a frame or a region in the viewer, where the idea shows up.

​

What you need

• The ChArUco board, printed at 100% scale on US Letter or A4 paper. The board is the same on both. The bar under the board must measure 50 mm.
• A ruler, to measure one printed square and one marker, and three dimensions of your object.
• A kitchen scale, for the object’s mass.
• A hardback book or a stiff board to use as a ramp.
• A phone app that reads tilt in degrees. On an iPhone, use the Level tab of the built-in Measure app. On Android, install Bubble Level.
• A pencil or the edge of a table, to find where the object balances.
• A desk lamp that you can move, for the photo under a second light.
• A phone that records video with exposure and focus locked.
• A Google account for Colab and Google Drive, with a T4 GPU runtime.

Stage 1 explains how to film the orbit. Stage 5 explains the measurements that go into capture/physical.json: the mass, five tilt angles at which the object starts to slide, three dimensions, three balance readings from a named end of the object, and, if your object is made of a metal part and a non-metal part, the two materials. Stage 6 explains the photo under the second lamp.

​

Choosing your object

Pick one object that is:

• rigid and opaque, so no glass, cloth or plush toys;
• not very shiny, since a mirror-like surface defeats both the pose solver and the material model;
• small enough to fit within a Letter sheet next to the board;
• light enough for a kitchen scale and heavy enough not to blow over on the ramp;
• textured, with print, grain or marks on its surface. If it is plain, add a few stickers or pencil marks.

AllSides excludes transparent and deformable objects too.

​

Weekly checkpoints

The project runs for five weeks. Use these checkpoints to pace your work. Film your object in the first days of week 1. If stage 2’s quick splat looks wrong, film again right away, while a reshoot costs you only an afternoon.

​

Stage notebooks

​

What you hand in

Your submission has two parts, because some of the files are too large for a repository. In a projects/real-to-sim-twin directory of your class repository, commit:

• the six stage notebooks, executed on your own object, with every cell output saved and your answers in the answer cells;
• the two scorecards from stage6/, scorecard_appearance.json and scorecard_behavior.json;
• a submission.md that names your object, links your Drive project folder, lists any stage where you fell back to the reference with the reason, and holds your shortfalls section.

Keep the large files in your Drive project folder: the merged recording stage6/pipeline.rrd, the USD asset with its textures in stage6/asset/, the capture video and frames, the held-out frames, and the photo under the second lamp. Share the folder view-only with the course staff and paste its link in submission.md. The graders read the folder’s modification times, so do not change anything in it after the deadline. Keep the recording under 500 MB, so that it opens in a browser, and the asset with its textures under 200 MB. Stage 6 checks both limits. The shortfalls section is the last part of your write-up. Name the two or three places where your twin differs most from your object, give the number that shows each one, and name the stage it comes from. Stage 6’s last question walks you through it. If you fell back at any of stages 1 to 4, your stage 5 and 6 numbers describe the reference object, so also give your own measurements (mass, tilt angles, dimensions and balance readings) beside the reference ones. Submit the full GitHub URL of the directory to Canvas.

​

When a stage fails on your capture

If one of stages 1 to 4 fails on your capture and you cannot fix it by filming again, you may continue from the reference object. When a stage’s check fails, it writes nothing, and the next stage finds its input missing and falls back to the matching reference intermediate. Your own outputs stay in the project folder, renamed with a .yours suffix. A fallback costs you the output half of that stage’s points. You keep the answers half: answer the stage’s questions with your own failed run and explain why it failed, and that explanation is graded as one of the answers. The later stages then run on the reference object and are graded normally, on the reference. You may fall back on at most two stages. With three or more, the object is no longer yours, and the project is capped at 50 points. Stages 5 and 6 have no fallback of their own, because they need your real object on the scale, on the ramp and under the second lamp. After a fallback at any of stages 1 to 4, they run entirely on the reference object, including its measurements, because your measurements do not fit the reference geometry. Your own capture/ folder is kept as capture.yours/. Report your measurements beside the reference ones in the shortfalls section.

​

References

• Hartley, R., Zisserman, A. (2004). Multiple View Geometry in Computer Vision.
• Kerbl, B., Kopanas, G., Leimkuehler, T., Drettakis, G. (2023). 3D Gaussian splatting for real-time radiance Field rendering. ACM Trans. Graph.
• Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., et al. (2023). Segment Anything. arXiv [cs.CV].
• Omniverse, N. (n.d.). Closing the real-to-Sim-to-real loop for physical AI.
• Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., et al. (2025). VGGT: Visual geometry grounded transformer. arXiv [cs.CV].
• Wu, Q., Esturo, J., Mirzaei, A., Moenne-Loccoz, N., Gojcic, Z. (2024). 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting.
• Ye, V., Li, R., Kerr, J., Turkulainen, M., Yi, B., et al. (2024). gsplat: An Open-Source Library for Gaussian Splatting.

---