Articulated digital twins from monocular video

I built a pipeline with GPT-6 Astra to reconstruct articulated 3D assets from an RGB walkthrough of our lab kitchen at MIT CSAIL.

The geometric input was a handheld phone video of about 20 seconds, without depth-sensor measurements, supplied CAD models, or retrieved assets. The workflow took roughly one day of elapsed time, including overnight downtime and human intervention.

Live: frank-zy-dou.github.io/kitchen-twin

Thin structures, reflective surfaces, several unfinished assets, and the room shell still require revision.

Reconstruction and verification

• ViPE estimates camera poses and near-metric depth from the monocular video, providing a geometric reference for modelling.
• GPT-6 identifies the objects to model. Open-vocabulary detection locates them, and per-object masks are tracked across frames. Observations are fused in 3D, with dimensions estimated in metres from the reconstructed depth.
• For every asset, GPT-6 works in its own sandbox with tools: it writes the object as a program in a small Blender DSL (closed solids, PBR colours, hinges/drawers), builds it, renders it over the original video frames, compares it with the scan points in 3D, and iterates.
• A separate GPT-6 session reviews each asset against video frames and supports its revision requests with those frames.
• A completeness pass renders the whole modelled scene from the video's own cameras, puts it next to the real frames, and says what is still missing. What the detector keeps missing (a row of identical cabinets) gets placed geometrically instead.
• The pipeline exports MJCF/URDF models with joints.

From geometry to mechanics

The exported joints specify kinematic structure, but do not establish physical validity. A drawer can look right in a render and still have the wrong travel limits, friction, or collision geometry. Visual agreement constrains appearance and aspects of geometry; the mechanical parameters needed for interaction require separate evidence.

What visual verification can establish

I use agreement with the video as evidence of reconstruction quality, not as a measurement of every physical property. Surfaces hidden throughout the walkthrough remain underconstrained; a plausible completion should be marked as an assumption. Camera and depth estimates also provide a shared reference, so agreement between a model and its scan may preserve errors in that reference.

The separate verifier adds a review stage, but its judgments still rely on the same video and estimated geometry. A separate model session does not supply independent ground truth.

From reconstruction to system identification

An articulated twin could carry uncertainty alongside its geometry: which surfaces were observed, which dimensions came from estimated depth, and which joints remain hypotheses. A robot could then choose interactions that reduce uncertainty relevant to its task, such as probing drawer travel before planning a pull. This would connect visual reconstruction with active system identification.

My original experiment on X.

ViPE: Huang, J., et al. (2025). ViPE: Video Pose Engine for 3D Geometric Perception. NVIDIA Research Whitepapers.

@misc{dou2026kitchentwin,
  author = {Dou, Zhiyang},
  title = {Articulated digital twins from monocular video},
  year = {2026},
  month = sep,
  howpublished = {Research Notes},
  url = {https://frank-zy-dou.github.io/blog/kitchen-twin-from-video/}
}