Interfaces shift from calling prebuilt pages to generating the next frame

Today's software stores structured state, and code renders that state into buttons, text, images, and animations. When you click a button, an event handler executes rules a developer wrote in advance, changes the state, and the interface re-renders. Even when AI generates an entire webpage from a prompt, the final deliverable is usually HTML, CSS, JavaScript, or some kind of component tree.

That traditional path pre-writes possible states and actions into the system, which means the same state can be read, searched, audited, and reproduced, and permissions and errors can be handled within a structured framework. Behavior the developer never defined won't just appear, and visual design has to be translated into elements that code can express. Subtle materials, spatial relationships, and continuous motion often get lost in that translation.

Solaris changes that last stretch. The current frame is no longer just the output of code; it also becomes the state the model uses to predict the next frame. Clicks, drags, and text instructions are fed into the model along with the image, and the next frame is generated on the spot. Runway describes this as "no intermediate representation": interface behavior in a demo no longer needs to be translated item by item into pre-written front-end rules, while model services, data connections, permissions, and underlying computation stay in the system.

As a result, the same thing can now be produced in two different ways. Traditional software writes all possible states into code ahead of time, while Solaris generates more on-screen change in the moment of interaction.

How clicks get into a video model

Solaris builds on Runway's Gen-4.5 video generation model and follows the GWM-1 approach. A world model predicts what comes next given the current environment and an action. GWM-1 already generates frame by frame and accepts action conditions like camera pose, robot commands, and audio. Solaris turns that action conditioning toward mouse, touch, and text input.

When a user clicks on a shirt, the system does not look up a fixed event for a "shirt button." Instead, the current frame, the click location, and the interaction semantics become the conditions for the next frame. The model learns visual consequences from the actions it has already seen, so the same drag gesture can move an object, change a material, or trigger a scene transition depending on the context.

A standard video diffusion model runs dozens of denoising steps across the entire clip, and a wait of several seconds or minutes is acceptable for content production but useless for continuous interaction. Runway made three changes to get Solaris to real-time speeds. First, it switched to frame-by-frame autoregressive generation, where each frame depends only on what's already been produced. Second, it distilled the many denoising steps down to a few. Third, the fast model keeps learning from its own output to reduce quality drift over longer generations.

The language model and the world model split the work inside the loop. The language model interprets what the user asked for, decides whether the current scene should change in place or switch to a new one, and produces a prompt that guides Solaris. Solaris then draws that decision into a continuous sequence of operable frames. Separating reasoning from rendering means the language model doesn't have to rebuild every pixel in code, and the video model doesn't have to understand the whole task on its own.

Runway says the experience starts to feel less than real-time once interaction latency approaches half a second. The research page states that Solaris reaches interactive speed, is orders of magnitude cheaper than a standard video diffusion model, and maintains 720p quality.

What the four demos reveal about the new interactions

The Solaris demos fall into three categories of change: interfaces become fully visual, scenes persist and stay alive, and the range of interaction is determined by what the model can do.

In the virtual try-on demo, the reference image of a person, the rack, and the shirt all sit in the same scene, and the user drags the clothing directly onto the figure. The salad demo has no fixed pipeline of "ingredient list, confirm, generate result"; the act of dragging ingredients into the bowl is itself the operation. In the living room scene, lighting, furniture, and materials change across continuous frames, and the interface doesn't just swap in a fresh static page after a click. The fire-safety lesson lets users drop different materials into the flame and watch the model generate the ensuing reaction.

The four scenarios span try-on, spatial design, food combination, and science education. The same generation loop doesn't require switching between different menus, forms, and confirmation pages. The objects, positions, and continuous changes within a scene are the interaction language.

What the two evaluations each show

Runway first measured how much information is lost when an interface is rebuilt from a screenshot. GPT-4o, Gemini 2.5 Pro, and Claude Fable 5 each received a single screenshot and had to reconstruct the corresponding interface. The evaluation used DINOv3 feature similarity, CLIP similarity, and structural similarity across ordinary webpages, slides, graphic designs, and natural scenes. Scores for all three models drop as the visual content gets more complex, with Fable 5 posting higher overall scores and natural scenes producing the lowest reconstruction scores.

1. Reconstructing interfaces from screenshots: is information lost? GPT-4o · Gemini 2.5 Pro · Claude Fable 5 | DINOv3, CLIP, structural similarity

How to read it: as the source becomes more visually natural, all three models score lower on reconstruction; Fable 5 scores higher overall.

The results show that squeezing a rich visual scene into language or code and then rebuilding the interface loses information steadily in layout, materials, and spatial relationships. Solaris keeps working directly in pixel space to continue the current frame, precisely to avoid that compression-and-reconstruction round trip.

The second experiment compared Solaris against coded interfaces generated by Claude Opus 5. Both started from the same image and the same interaction request. 250 participants evaluated 30 examples, producing about 7,500 pairwise judgments on which result followed the instruction more closely and which felt more natural in the scene.

Two side-by-side comparisons of Solaris and coded interfaces Each video shows the same interaction command applied to both sides

Click any element, and it keeps its shape while slowly floating upward.

The camera moves smoothly right and down from the lighthouse balcony, ending on coastal waves.

How to read it: the chart shows 62% prefer Solaris for instruction following, and 72% for scene naturalness.

The preference chart aggregates judgments from 250 participants—about 7,500 in total: 62% chose Solaris as better following the instruction, and 72% said Solaris looked more natural in the scene. Two side-by-side videos illustrate what participants actually saw. In one, clicking an underwater object keeps its shape while it floats upward. In the other, the camera moves continuously along a balcony, across the coast, and over the waves in a lighthouse scene.

How the generation model and existing software split the work

Solaris changes interface rendering from "code calls a prebuilt component" to "the model generates the next frame," and that introduces a new division of labor in the software stack. The generation model handles materials, spatial relationships, continuous motion, and open-ended visual responses. The structured systems handle identity, permissions, product data, transactions, logs, and searchable text.

Runway also describes a multi-model collaboration: a language model understands intent and decides when the scene should change, an image model handles crisp text and static visuals during pauses, and a video model produces continuous motion. Product data, documents, and account state come in through structured services, and the three model types together form an interface that stays operable.

How Solaris trains AI agents

Solaris has another purpose that isn't aimed directly at everyday users: training computer-operating agents. Today's agents often memorize button positions on fixed webpages, and their success rate drops sharply when a hotel site changes its layout, a pop-up covers the target, or the information refreshes mid-task.

Solaris can keep generating layouts and interactions the agent has never seen, giving it far more visual variation within the same task. The training objective can also shift from "find the coordinates and click" to "understand the objects, actions, and consequences in a scene." This is the same idea behind training robots with world models: first experience variation in an environment that can be generated at scale, then transfer the ability to real systems.

OSWorld 2.0 demonstrates the scale of what such training is up against. The benchmark contains 108 long-horizon real-world workflows, each with a median human completion time of about 1.6 hours. On the binary 500-step completion metric, the best system reported in the paper finishes only 20.6% of tasks. These tasks constantly switch pages, pop-ups, and intermediate states, making continuously generated interfaces a useful way to expand the variation seen during training.

In the training setup, structured services record the task goal, correct states, and feedback, while Solaris generates the layouts, objects, and visual changes. The same task can therefore quickly produce different button positions, pop-ups, occlusions, and scene refreshes, so the agent practices understanding objects, actions, and consequences instead of memorizing coordinates.

Which products Solaris might change first

The official demos point to four direct use cases:

• Interactive prototyping. A concept image can become a clickable, draggable, continuously changing interface for presenting design ideas and brand spaces.
• Shopping and configuration. Clothing, furniture, materials, and products can be combined directly in one scene, with the selection process and the visual result happening at the same time.
• Teaching and exploration. Learners can move objects, change conditions, and watch continuous reactions, turning an explanation into a scene they can operate.
• Agent training. The same task can generate a wide range of layouts and process variations, giving computer-operating agents a constantly changing practice environment.

Solaris pushes the video model from content generator to real-time interface engine: it continuously accepts human input and generates the next operable frame on the spot. Interface rendering changes from a one-shot coded output to a continuous generation process, and the video model begins to take on the display layer and part of the interaction layer of software.