How the brain sees a moving world: a gap still unfilled

The brain's visual system has a well-defined division of labor — classically described as two pathways: the ventral stream for recognizing objects, and the dorsal stream for handling motion and action. But more recent research has found a third: one dedicated to processing social information, and currently the least understood.

In the past, studying what a given brain region prefers to look at meant hand-picking or hand-designing static images — for instance, showing a face-selective region a pile of faces. In recent years, researchers went further, using deep neural networks as encoding models to predict the brain's response to images, and even synthesizing images that maximally activate a given region. But nearly all of these methods stop at static images, while human vision is fundamentally dynamic — dodging obstacles, tracking objects, reading others' intentions all depend on continuously changing frames. The search space for video is far larger than for images, and dynamic vision has lacked a systematic method for generating and evaluating stimuli. That's the gap NEvo sets out to fill.

First, build the brain a digital stand-in to rehearse on

To evolve a video that lights up a given brain region, step one is building a model that can predict the brain's response — the authors call it a digital twin: feed it any video, and it predicts how excited each small patch of the visual cortex will get.

Technically, the backbone network is V-JEPA 2 (Meta's self-supervised video understanding model). Rather than taking its top-layer semantic features, it extracts several intermediate layers (attention features from blocks 13, 15, 17, 19, 21, and 23), averaged across time and space. Then, for each voxel (the smallest 3D spatial unit in fMRI — essentially a "pixel" of the brain), it runs a separate ridge regression (a statistical method for mapping model features onto predicted brain-response values), mapping the model's features onto the real fMRI response. Each voxel gets assigned to whichever layer predicts it most accurately on the validation set — effectively aligning different locations in the brain with different depths of the model. Training data came from two real human fMRI datasets: BOLDMoments (short video events) plus a social-interaction dataset.

An ablation study confirms this design choice: the encoding model built on V-JEPA 2's multi-layer features clearly beats the common CLIP baseline (CLIP extracts per-frame image features), and multi-layer features beat any single layer — showing that a genuinely time-aware video representation combined with layered encoding really matters.

Step one: build a digital-twin model that predicts the brain's visual responses. Source: EPFL NEvo project page.

Break a video into genes, then run a brain-scale survival of the fittest

With a scoring digital twin in hand, the next step is to search for the video that produces the highest score in the target brain region. NEvo breaks a video into a string of genes: subject, lighting, motion, mood, event structure, camera movement. Each gene picks one option from a category of attributes, strung together into a comma-separated text prompt fed to a video generation model.

Controlled experiments show this evolutionary search clearly beats random search, and also beats BrainDiVE, a comparison method that optimizes directly via gradients. BrainDiVE flat-out fails in this dynamic-video setting, producing stimuli whose activation is only 24% of the average natural-video level; NEvo beats it at every data point. The authors' explanation: evolutionary prompt search keeps synthesis constrained to videos that are semantically meaningful and temporally coherent, which is more reliable than gradient-based methods.

Step two: treat videos as a gene pool, evolving prompts through selection, crossover, and mutation. Source: EPFL NEvo project page.

Specific parameters: population size N=20 per generation; the top 30% by activation are kept as parents for the next generation (elite selection); crossover rate 0.5, swapping gene fragments between two prompts; mutation rate 0.2, randomly resampling an attribute. The genetic search's advantage over random search reaches Cohen's d=0.46 (p<0.001), and it's also more stable than hill climbing (a simple search method that only takes small steps in a better direction) in this high-dimensional prompt space.

Lock in the picture first, then make it move

Searching content and motion simultaneously is too expensive, so NEvo splits the job into two steps: first search for a static image that best activates the target region as an anchor, then — once the picture is locked — search only for how to make it move.

In stage one, the fast image generator SDXL-Turbo (single-step, 512×512) searches static content — subject, scene, texture, lighting — to find the image with the highest predicted activation as a visual anchor, spending 400 evaluations. In stage two, the anchor image is fixed, and the search covers only dynamic attributes: motion type, pacing, camera movement, event structure, interaction type — using the image-to-video model LTX-Video 2.3 (a 22B distilled diffusion transformer with Gemma-3 12B as the text encoder, producing 256×256, 24fps, 2-second clips) to animate the anchor, spending 200 evaluations. Ablation studies confirm the two-stage approach beats single-stage search (searching only images or only videos).

The benefit of the two-stage split is clear in the fusiform face area (FFA, which handles face recognition), since it needs to first find a suitable face as an anchor before optimizing motion; the benefit is less clear in the motion area MT, since MT is already driven broadly by rich motion and doesn't care much about the specific picture.

Step three: lock in the strongest static image first, then make it move. Source: EPFL NEvo project page.

The evolved videos suit the brain's taste even better than reality does

Tested across representative brain regions — FFA, PPA (scene area), EBA (body area), MT, V3A, and pSTS — NEvo's synthesized videos outperformed the strongest of a large pool of real videos and professionally designed stimuli.

And it successfully reproduced each region's known preferences: face-like content emerged for FFA, scenes for PPA, bodies for EBA, coherent motion for MT/V3A, and interactive dynamics for pSTS.

Motion itself also lights up the brain further — it's not just about the picture. Comparing the optimized video against its own first static frame, the moving version activates on average a fair bit more:

The motion area MT shows the biggest gain, and even FFA — traditionally considered a static-category region — shows a measurable improvement. Temporal structure broadly shapes visual selectivity; a lot of brain regions really do prefer things that move.

Videos tailored to each brain region: FFA produces faces, PPA produces scenes, EBA produces bodies, MT produces motion, pSTS produces social scenes. Source: EPFL NEvo project page.

Walk the brain's lateral pathway, and vision turns social

This is the paper's most scientifically valuable finding. Researchers ran a searchlight (an analysis method that scans point by point across the cortical surface) from V1 all the way to the anterior superior temporal sulcus (aSTS), synthesizing the strongest stimulus for each small patch, then automatically captioning each synthesized video and clustering the captions into word clouds. The result traces a smooth gradient.

From low-level spatial structure, to body motion, to high-level social interaction — it's one continuously climbing pathway. The correlation between attributes and activation confirms this gradient too:

Moving up the lateral pathway, vision gets progressively more social: the searchlight scan plus word clouds reveals the gradient. Source: EPFL NEvo project page.

Two clay discs, pulled apart to reveal each region's taste

The cleverest control experiment: start from a completely unnatural image — two stacked clay discs — fix that image in place, and optimize only its motion.

From the same abstract starting point, optimizing for different brain regions forces out completely different characteristics, cleanly isolating what each region prefers. This pipeline — encoding model plus gene-encoded prompts plus two-stage search — gives neuroscience a new, hypothesis-driven tool for controllable stimulus generation: for any given brain region, you can evolve a customized video to specifically test a hypothesis about that region's preferences, without researchers having to hand-design thousands of candidate stimulus images.

Using abstract stimulus probes: starting from stacked discs, optimizing for pSTS grows face-like interaction, optimizing for MT produces pure motion only. Source: EPFL NEvo project page.

This is still a prediction, not yet a measurement

The paper is explicit about this: all of these activation scores, and which video is "strongest," are predictions made by the digital-twin encoding model itself — not brain activity actually measured after real humans watched these videos.

The paper positions itself as generating new, testable hypotheses for in-vivo experiments; in the discussion, the authors call for future closed-loop brain-imaging studies to verify whether the videos NEvo creates actually evoke the predicted brain responses — this is the work's biggest caveat. Beyond that, conclusions drawn from synthesized stimuli are subject to the encoding model's own biases, and both the encoding model and the video generation model may carry bias. The current system is purely visual, with no sound, which limits its ability to characterize regions like aSTS, where communication often depends on speech or audiovisual cues. The structured prompt space improves interpretability, but may also miss visual dimensions the schema doesn't cover.

This system, called NEvo, works by letting AI "evolve" videos on its own. First, build the brain a "simulator" that predicts how excited any given brain region will get from any given video — the authors call it a "digital twin." Then run a method that mimics natural selection: cull and breed, generation after generation, forcing out the video that lights up the target region the most. To save on compute, it's also split into two steps: first quickly search out the strongest static "base image," then — once the picture is locked — optimize only "how to make it move."

Break a video into a string of "genes" (subject, lighting, motion, mood…), then breed it like livestock: generate a batch → have the brain simulator score each → keep the strongest 30% → crossover plus random mutation to breed the next generation. The score keeps climbing until it evolves the video that best lights up the target region.

The most valuable finding is a "hidden map of the brain." Researchers used a "searchlight" — like scanning a flashlight point by point across the brain's surface — sweeping from the most basic visual area all the way to the front of the brain, letting each small patch evolve its own strongest video, then seeing what each patch loves to watch. The result traces a smooth transition zone.

The further forward you go, the more "social" vision becomes: from preferring texture and color, to preferring motion and bodies, to physical interaction, and finally face-to-face social contact.

There's also a clever piece of corroborating evidence: take the same completely abstract image that doesn't resemble anything — two stacked clay discs — hold it fixed, and change only how it moves.

The same starting point, "forced" into two completely different preferences — cleanly isolating what each brain region is drawn to.

Line up tens of thousands of real-world videos by how strongly each "lights up" a given brain region — where would AI's out-of-thin-air evolved video land in that lineup?

Also, the act of making the picture "move" lights up the brain further on its own: the optimized video activates on average a fair bit more than its own still first frame (the video's very first frame) — most notably in the motion area (nearly doubling) — and even the face-recognition area, traditionally thought to only care about static content, shows a measurable improvement.