MirroS researchers are arguing that video is the wrong format for teaching AI about physics. Their Code-as-World project converts footage of real scenes into executable programs that the MuJoCo physics engine can run, turning each clip into something a machine can edit, verify, and re-simulate.
The team’s claim is that pixels only evidence a scene. A video model can generate plausible frames without ever encoding mass, contact, or gravity, so the project instead rebuilds scenes as scene.json files and uses an agentic loop that recovers them from real footage in up to five rounds, checking each reconstruction against the original video.
The payoff is supervision. Verified worlds carry exact physical labels that raw footage lacks, and training on them produces Code-as-World-VL-9B, which scores 55.4 MRA on the project’s evaluation.
For robotics and physical AI, editable world programs let an agent test an action before taking it. MirroS is not alone in that direction, but making code, rather than learned latent space, the substrate of world models is the distinctive bet here.
