Why traditional video understanding was expensive and error-prone

The logic of static processing is straightforward: the developer decides on a fixed sampling rate, and the system processes the whole video at that rate. Gemini's default static setting is 1 FPS (1 frame per second); developers can adjust this via the API.

The issue with a fixed frame rate isn't that it's inherently wrong, but that it treats the entire video uniformly. A 90-minute presentation at 1 FPS yields 5,400 sampled frames. If you increase the rate to 5 FPS to capture faster motion, that number jumps to 27,000 frames. The actual token count also depends on resolution and encoding, so you can't simply equate frames with tokens, but the pressure on context and cost from increasing frame counts is real.

Meanwhile, 1 FPS isn't truly comprehensive viewing. If a button changes from 'off' to 'on' for only 0.3 seconds, that change could easily happen between two samples. The model never sees that frame, and no matter how powerful its reasoning, it can't recover a visual fact that never made it into the context.

Developers were thus forced to choose between two risks:

• Higher FPS: Less likely to miss brief actions, but the entire video pays a higher cost for a few key moments.
• Lower FPS: Keeps costs manageable, but risks losing scene boundaries, fast action, flickering anomalies, or briefly appearing objects.

The static mode still has its place. For tasks under a few minutes, where low latency is critical, or where a uniform sampling rule must be applied across the entire clip, it's simpler and more predictable. The agentic mode addresses a different problem: when the video is long but the answer depends on only a few snippets, the model needs to decide for itself where to spend its observation budget.

How Gemini decides where to look

In the diagram Google published, the model cycles between 'thinking' and 'observing.' It doesn't unfold the entire video first and then start answering. During its reasoning process, it can invoke three types of internal tools: reading subtitles, reading audio from a specific time range, and fetching frames at a specified start/end time and FPS.

This cycle can be broken down into roughly five steps:

1. First, understand what evidence the question requires. 'What did the speaker announce?' needs a different signal than 'How many jumping jacks did the person do?'
2. Use low-cost clues to narrow the search. If subtitles are available, the model can search for keywords and timestamps; if it needs to hear tone or ambient sound, it reads the audio.
3. Select a specific time window. The model reduces tens of minutes or even hours of video to a few suspected clips, rather than maintaining high-density sampling across the entire length.
4. Choose the modality and frame rate based on the task. Language questions might rely mainly on subtitles, while scene cuts, state changes, and action counts require visuals, possibly with locally increased FPS.
5. Re-watch if evidence is insufficient. After acquiring a new observation, the model continues reasoning, and if necessary, switches time ranges, modalities, or increases the sampling rate until it can form an answer.

This same mechanism yields both savings and accuracy gains: by watching fewer irrelevant segments, token use drops; by concentrating more of the frame budget on key windows, brief details are more easily captured. Both benefits stem from this single observation allocation loop.

A point that's easy to misunderstand here is that agentic mode doesn't mean 'watch the video losslessly.' It still selects, samples, and narrows its focus, so if it picks the wrong time window, it can miss information. Its advantage is that it lets the model dynamically adjust its observation strategy based on the question, rather than pinning all hope on a single developer-preset FPS.

What the three video demos actually prove

Google's launch page features three side-by-side demo videos. They aren't three isolated features, but rather demonstrate three different types of observation allocation challenges: 'long-video cost,' 'high-speed action,' and 'long-video retrieval.'

1. LongVideoBench: Same long-video question, far fewer frames

The first demo compares Gemini 3.7 Flash's performance on LongVideoBench in static mode versus agentic mode. The key isn't that the model 'skips' the video; it's that it shifts from uniform sampling to a targeted investigation focused on the question: first locating potentially relevant segments, then concentrating its observation budget there.

LongVideoBench itself requires the model to handle long, multi-modal intertwined video content, making it well-suited to test whether the model can maintain event order, object relationships, and question clues within a long context. Google's demo emphasizes that agentic mode uses significantly fewer input tokens while delivering higher answer quality.

2. Jumping jack counting: Use higher FPS and re-watch when necessary

The second demo asks Gemini to count fast, repetitive jumping jacks. A fixed 1 FPS can easily compress the start and end of a single repetition into the same sampling interval, causing undercounting or misinterpreting a half-motion as a full one.

Agentic mode first locates the segment where the action occurs, then scans and re-watches those specific moments at different FPS rates. This example shows where 'dynamic FPS' truly shines: not in maintaining a high frame rate across the entire video, but in temporarily increasing observation density within a few critical seconds.

3. Needle in a haystack: Locate first, then answer complex questions

The third demo compares token consumption between the two modes when answering complex questions about a long video. Agentic mode can leverage subtitles, audio, or lower-density frames to find clues first, then concentrate high-cost visual reads on a few candidate segments.

This approach is especially well-suited for product launches, lectures, meeting recordings, sports footage, and multi-hour archives: the question usually concerns just one event, one sentence, or a briefly appearing object, so there's no need for every frame to enter the context at equal density.

How to read the three sets of measured data

Google's three data sets come from Gemini 3.7 Flash, comparing static 1 FPS against agentic mode. The tests uniformly use the high thinking level and low media resolution. Gains vary significantly across benchmarks, so focusing only on the 'up to 88% savings' headline misses the most important point: the structure of the task determines how much agentic observation can save and how much accuracy it can improve.

Three conclusions emerge from this table:

• The largest token savings don't correspond to the largest accuracy gains. 1H-VideoQA and LVBench both save 88%, but their relative accuracy improvements are only 1.1% and 4.1%, respectively.
• MINERVA shows smaller token savings but the biggest quality gains. These complex reasoning tasks require more effective observations; agentic mode can't just focus on watching less, it must also ensure sufficient evidence.
• The 'up to 7%' figure is a relative improvement. In the chart, MINERVA rises from 73.7% to 79.0%, an absolute gain of 5.3 percentage points, which is roughly a 7.2% relative improvement. These two metrics shouldn't be conflated.

Google also reports that across these tests, analysis costs were reduced by up to 66%. The cost reduction doesn't directly equal the token reduction because different modalities, reasoning processes, and tool calls all affect the final bill.

The 'Pareto frontier' here means that among the models tested, no other option can achieve higher accuracy without increasing cost, or lower cost without sacrificing accuracy. It's a relative position within that specific test set, not a permanent ranking across the entire video understanding market.

What real-world uses early partners describe

Google also features testimonials from four early partners. This is vendor-supplied data, not an independent benchmark, but it broadens the picture beyond what lab evaluations cover:

• Ponder uses Gemini to find usable clips from raw footage. It reports that, while maintaining similar recall, its new single-call workflow uses roughly 3.5 times fewer input tokens.
• Revyl uses cloud-hosted devices to test iOS and Android apps. It was concerned with catching fast animations and state changes that screenshot-based verification tends to miss.
• Resemble AI tested on about 1,000 annotated videos and real-time deepfake detection workloads, reporting a 57% reduction in input tokens alongside improved accuracy in long-video descriptions and context.
• Mosaic, in production video editing evaluations, reports a median input token reduction of 97% and nearly double the constraint coverage, noting that agentic mode was the only approach that completed its 4.5-hour podcast case.

These results show that enterprises aren't just paying for 'video understanding' per se, but for collapsing the search, segmentation, resampling, and review workflows that engineering teams used to hand-code into a single model interaction. However, these partner figures lack a common public testing protocol and can't be directly compared to the standard benchmarks above.

Which applications are best suited

Sub-second moment retrieval

Rough cuts, scene boundary detection, and UI state changes can all happen within 1 second. Agentic mode can first locate a coarse region, then increase the FPS within that window to narrow down to a more precise cut point. It improves the feasibility of automated editing, but Google hasn't published frame-level error distributions across video types, so it can't be universally described as achieving 'industrial-grade precision.'

Needle-in-a-haystack search in long videos

In 90-minute lectures, meetings, or multi-hour recordings, the model can search using subtitle and audio cues first, then verify against nearby frames. It's best suited for point questions like 'When was a certain sentence said?', 'When did a certain object appear?', or 'Which segment demonstrates a specific feature?'.

Anomaly detection

Anomalies in surveillance and quality-control footage often last only a few seconds. A model can first do a low-cost global scan, then re-sample suspicious time windows at higher detail to catch fast motion, flicker, or visual artifacts. But none of this amounts to the zero-miss guarantees that safety systems require—high-risk deployments still need dedicated detectors, explicit thresholds, and human review.

Action and object counting

Repetitive structures appear everywhere—exercise reps, parts moving down an assembly line, or sports movements. Locally raising the FPS recovers intermediate states that a fixed 1 FPS stream would miss, and the model can then review those windows to verify count boundaries. Counting can still go wrong when actions are poorly defined, occlusions are severe, or multiple objects cross paths.

Video production and asset management

For editing teams, a more practical payoff is turning "find three usable shots in dozens of hours of footage" into a natural-language query. The model narrows down candidate clips, and the editor does the final confirmation. This is far more realistic than letting the model perform the final cut itself, and it produces a much cleaner audit trail.

How developers call it

The capability is available through the Gemini API in Google AI Studio and the Gemini Enterprise Agent Platform, supporting both direct video uploads and YouTube URLs. It currently works with Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite, billed at standard Gemini token rates with no extra feature fee.

The key change on the calling side is adding "processing": "agentic" to the video input:

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3.7-flash",
    input=[
        {
            "type": "video",
            "uri": "https://youtu.be/7Z5Vy9JBANs",
            "processing": "agentic",
        },
        {
            "type": "text",
            "text": "What are the 3 most important announcements in this keynote?",
        },
    ],
)

print(interaction.output_text)

Developers can also inspect interaction.steps. If the response contains processing_call and processing_result, the model genuinely invoked the video processing tools to navigate dynamically, rather than just reading the footage in a static way. This distinction matters for cost auditing and debugging: setting the config field correctly does not guarantee that every request follows an identical tool path.

A simple heuristic can guide mode selection:

Where it still falls short

First, agentic mode reduces irrelevant observations—it does not turn video analysis into something free. The broader the question and the more segments that need verification, the more frames, audio, and reasoning steps the model may still invoke.

Second, active selection introduces new failure modes. If static mode can drop frames due to fixed sampling, agentic mode can misread search cues, select the wrong time window, or stop observing too early. For important work, check the tool receipts and keep sampling-based review or a second detection path in place.

Third, the three benchmark sets and partner cases do not prove reliability across every surveillance, medical, security, or production-QA scenario. The long-video benchmarks show the direction is promising, but real systems still have to contend with occlusion, low light, compression artifacts, subtitle errors, and shifting domain distributions.

Fourth, product rollout has a timeline. Google says the capability will soon cover Flash and Flash-Lite users in the Gemini App, and will reach Ask YouTube on YouTube watch pages in the coming months. These are release plans—not evidence that the feature is already fully available to all general users.

What this update really changes is how video enters the model. Previously, developers decided how much footage to show, and the model could only reason over the frames it already had. Now the model can treat "where should I look next" as part of its reasoning. For long videos, the context window stops being merely a passive container and starts to behave like a workbench—something you can search repeatedly, zoom into, and cross-check.