Microsoft Open-Sources Code Video Model: Generating Video with Code

Microsoft Research has introduced a new video generation framework: Code Video Model.

It makes AI video generation feel less like 'opening a blind box,' allowing creators to write code first to define scene layouts, motion trajectories, and physical dynamics explicitly.

Current video generation models (such as mainstream video diffusion models) operate much like a black box when generating video directly from text:

• Physical laws are mostly 'guessed': The model infers motion primarily from visual patterns learned during training rather than computing physical quantities like gravity and velocity step by step, making it prone to bizarre artifacts like clipping or anti-gravity motion.
• The process is difficult to control precisely: How the camera pans, how objects are arranged, and where paths lead are hard to foresee before generation using text prompts alone. If something is unsatisfactory, changing a single detail often requires re-rolling the entire video.

Key Capabilities of Code Video Model

By introducing an inspectable and editable code previs stage prior to final generation, Code Video Model provides four primary capabilities:

Scene layout and object trajectories can be directly modified. The program written by the AI places objects at specified coordinates and defines their movement paths over time. To change a character's walking route or adjust the relative spacing of furniture in a room, creators can directly edit the corresponding coordinate parameters and verify the result in the preview first.

It also supports localized modifications within the same scene. The paper demonstrates changing an archway into a doorway, swapping a humanoid robot for a quadruped robot, and switching a path from the left side to the right. Both the program and reference images are modified based on the original assets, aiming to alter only designated factors while keeping unrelated environments, camera work, and timing consistent. The final video, however, may still exhibit unintended variations.

Swipe horizontally to view the full diagram, or click below to open high-resolution image.

Camera motion and scene actions are planned independently. Camera dolly, pan, tilt, truck, and orbit movements can be controlled separately via code, independent of the object's own movement. For example, a subject can be frozen on a specific pose while the camera continues to rotate smoothly around it.

Incorporating physics equations as motion references. In scenarios that can be mathematically described—such as spring vibrations or double-pendulum motion—the program can compute motion according to corresponding equations, providing a physical reference for subsequent generation. For instance, a double pendulum consists of two interconnected rods whose motions influence one another. The program calculates their angles and positions over time based on coupled dynamical equations, rendering the result into proxy motion. Note that equations only constrain the code previs; one must still verify whether the final video faithfully follows these motions.

Preview before generating, without retraining base models. Creators first inspect a simple 3D preview, fine-tune motion and camera angles, and then pass it to the video model to generate materials and lighting. The paper uses off-the-shelf MiniMax-H3, guiding generation simply by altering the timing of proxy conditioning while keeping model weights unchanged.

Core Mechanism: Scripting Code Like Filmmaking Previsualization (Previs)

This technique borrows from the previsualization (previs) workflow in the traditional film industry: using simple 3D scenes to rehearse camera angles, actor blocking, and pacing before committing to principal photography and post-production.

Code Video Model adapts this industrial workflow to AI video generation:

Swipe horizontally to view the full diagram, or click below to open high-resolution image.

1. Building the Scaffold with Code: Rapid Preview, Direct Modification

Given a text prompt, a coding agent (GPT-6 in the paper) first writes an executable Three.js program.

After writing the code, the agent renders keyframes and frame grids covering the entire video, inspecting composition, occlusion, motion, and temporal progression against the prompt before modifying the code and re-rendering. While code generation and revision can be handled by AI, creators can also directly fine-tune parameters within the program.

This code explicitly acts as the 'scene builder and camera director': it places geometric primitives in 3D space, defines object displacements, plans virtual camera trajectories, and calculates mechanical dynamics via mathematical equations when needed.

The paper reports that rendering this script takes under one minute to produce a low-fidelity proxy video, serving as the proxy preview. Next, the system extracts a representative frame from the proxy video and prompts a large image model (such as GPT-Image-2) to redraw it into a reference image featuring the target lighting, texture, and photographic realism; users can also directly upload their own style reference images.

The benefits of this approach are straightforward: fast turnaround and what-you-see-is-what-you-get. If a table's placement or a vehicle's speed is unsatisfactory, tweaking the corresponding coordinate or velocity parameters in the code allows for immediate re-rendering and re-inspection. Simple parameter adjustments may involve just a single line of code. Note that 'under one minute' refers only to proxy rendering and does not include the time for AI code generation, reference image creation, or final video diffusion.

The accompanying browser viewer allows users to scrub through the timeline, inspect frames individually, and view the source code. Creators can also freely orbit, pan, or zoom the 3D perspective, toggling grids, camera trajectories, and the camera frustum (the spatial volume visible to the camera) to inspect spatial relationships between objects, trace the camera's path, and see exactly what will be captured.

2. Solving the 'Plastic Look' Dilemma: Early-Step Conditioning

Once a low-fidelity proxy video is ready, how do you turn it into a realistic final video?

A critical challenge arises here: if the proxy video is fed into the video diffusion model as a conditioning signal throughout all sampling steps, the model easily copies its crude geometry and flat shading, leaving the final output with a persistent 'plastic look.'

To resolve this, the research team proposed a training-free scheduling mechanism: early-step conditioning.

The team's experiments revealed that the proxy exerts different effects across generation stages:

• Early sampling steps: The proxy primarily helps establish spatial composition, camera transitions, and subject motion;
• Late sampling steps: Continuing to reference the proxy easily bleeds its rudimentary appearance into material and texture details.

Based on this characteristic, the team designed a temporal handoff: taking the default 20-step denoising schedule as an example, the system injects the low-fidelity proxy video as conditioning only during the first 5 steps (the initial 25%), allowing the model to quickly anchor spatial composition and motion scaffolds. Starting from step 6, the proxy video is removed, and the model continues generating materials, lighting, and fine details guided solely by the reference image and text prompt. The reference image and text prompt are maintained throughout the entire process.

The video model also fills in geometric and joint articulation details omitted by the proxy. Consequently, the initial script does not need meticulously crafted 3D assets; it only needs to convey the primary structures and motions clearly.

Here, the 'first 5 steps' refer to the first 5 computational iterations when generating the entire video—with each step updating the full temporal sequence—not the first 5 seconds on the video timeline.

The first 5 steps represent the default configuration used in the authors' experiments and can be adjusted in practice. In their tests, extending proxy conditioning generally made the final video adhere more closely to the original structure but risked retaining the proxy's aesthetic; shortening it weakened structural constraints. This represents an adjustable trade-off between control fidelity and visual realism.

Practical Applications: What Can It Do?

This division of labor is particularly well suited for production tasks requiring iterative adjustments to motion, routes, and camera angles. The three comparisons below come from the project's official demonstrations.

1. Bullet Time: Freezing Action While the Camera Keeps Orbiting

In classic cinematic 'bullet time' shots, two conditions must be met simultaneously: the subject's action freezes instantly while the camera continues smoothly orbiting and dollying around them. Using text prompts alone makes it notoriously difficult to specify exactly when motion should freeze and how far the camera should rotate thereafter.

In code, physical time and camera time can be decoupled into two independent variables and computed separately.

The code freezes the motorcycle and rider in world coordinates at a specific frame while continuing to drive the virtual camera around the rider. Upon receiving this proxy, the video diffusion model uses its motion and camera trajectory as guidance to render richer road reflections, vehicle finishes, and scene details.

2. Modifying Robot Trajectories: Adjust Paths First, Inspect Results Next

In robotics simulations, users frequently need to iterate on navigation plans for specific environments. If one merely edits text prompts and regenerates, scene details outside the trajectory often shift unpredictably as well.

With code as an intermediate layer, modifications become unambiguous. To alter a robot's turning path, creators simply open the program and adjust a few coordinate values in the waypoint list (route=[v(-0.75, 6.75), ...]), re-rendering a new preview video in under a minute. Once the trajectory is verified, it is passed to the video model for final rendering.

In the case below, the code previs plans a humanoid robot walking toward a laboratory doorway, while the final video complements it with realistic interior lighting, specular metal highlights, and mechanical joint details.

3. Architectural Walkthroughs and Product Cinematography

In interior walkthrough shots with complex spatial layouts, creators need to arrange furniture placement while ensuring the camera navigates through the space coherently.

A Three.js program can explicitly define the positions of sofas, stairs, and windows while scripting the camera's path through them. Reviewing the proxy allows creators to verify whether the camera is approaching the sofa, elevating its angle, or panning toward the stairs before deciding on final materials and lighting.

The same logic applies to product cinematography: for example, when shooting a sports car, the code can script a camera transition from an exterior wide shot to a wheel close-up. Once the proxy establishes the line of sight and perspective shifts, the video model fills in the glossy reflections on metallic paint and intricate mechanical textures on wheel rims.

Benchmark Evaluations: Balancing Structure and Realism

To systematically evaluate generation performance, the research team constructed CodeVideoBench, a 22-case benchmark spanning eight application categories: scene editing, robotics simulation, trajectory variation, gaming, product cinematography, architectural cinematics, physical laws, and bullet time.

The evaluation revolves around three core dimensions:

• Structure score: How closely spatial layout, subject motion, and camera trajectory in the video match the code proxy;
• Style score: Whether materials, lighting, and photographic texture align convincingly with the reference image rather than retaining the proxy's plastic look;
• Quality score: Temporal consistency and motion naturalness, checking for flickering, tearing, or artifacts.

Compared to full-step R2V—which also uses MiniMax-H3 but conditions on the proxy throughout all steps—Code Video Model achieved a higher overall score in the authors' tests. Under the Qwen-27B automated judge, the two scored 85.17 versus 75.22.

Even more telling of the trade-off are human blind evaluations: Code Video Model scored 6.92 in structure, lower than full-step R2V's 7.81; but scored 8.46 in style, higher than R2V's 6.02. Its overall score reached 8.17 compared to 7.64 for R2V.

In other words, withdrawing the proxy early brings the final video closer to the reference image's appearance, but at the cost of some alignment with the original structure. Its primary advantage lies in the balance between controllability and visual quality, which does not imply that every motion is strictly more precise.

These scores stem from the authors' self-constructed 22-case benchmark; the human blind study does not disclose participant counts or statistical significance, and conclusions apply specifically to this experimental configuration.

Practical Boundaries and How to Experience It

While code previs allows exact parameter specifications, the final video is still generated by a video model, meaning layout, pose, and motion may still deviate. When deployed for tasks demanding strict physical precision or frame-by-frame alignment, the final video must still be independently verified.

Reference images must also match the proxy. In the paper, pairing a humanoid robot proxy with a dog reference image resulted in an extra dog being generated alongside the original subject. Changing an interior reference image to entirely blue tones led the final video to revert largely to warm daylight, retaining only subtle blue accents. While code and reference images convey creator intent separately, the model may not always harmonize them as requested.

How Developers Can Get Started

The currently available entry point is the GitHub repository, which includes the codebase, pre-built examples, and evaluation benchmarks.

Developers can start with examples such as sports_car, tank_boarding, and loft_tour. These examples come with prepared prompts, proxy videos, and reference images, allowing users to skip code generation and proxy rendering to test the latter half of the video generation pipeline first.

Running locally requires the MiniMax-H3 model and an NVIDIA GPU environment; the repository recommends Ubuntu 22.04 and drivers compatible with CUDA 12.6. Running the end-to-end pipeline from text prompts also involves GitHub Copilot CLI for code generation and API calls to image models. The repository does not provide measured minimum VRAM requirements or end-to-end runtime estimates, so it should not yet be treated as an out-of-the-box, one-click video tool.

The report also showcases examples guiding Seedance 2.5 with code proxies, demonstrating that this previs concept can potentially be extended to other video models. However, these examples do not demonstrate the ability to manipulate the same 'first 5 steps' scheduling over closed-source APIs.

What makes Code Video Model noteworthy is that it provides creators with a visible, editable, and re-runnable shot arrangement before committing to final generation. Confirming 'how objects move and how the camera shoots' in code before letting video models realize the target visual texture provides a far more concrete foundation for iterative video editing.