How OMAC picks which frames and sounds to keep

First, the official animation — 37 seconds, three points.

Three names show up in that animation, and it's easy to mix them up — here's what each one actually is.

The common approach to compression is uniform pruning: downsample every frame equally, sparsify every audio segment equally. The problem is that video is full of redundancy — the same background can repeat for dozens of frames — while audio might have only a split second that's actually decisive. A blanket cut wipes out the redundant background and the key evidence together.

OMAC picks its priorities in four steps.

OMAC's four steps. On the left, the outputs of the video, audio, and text encoders feed into OMAC; the green box on the right runs, left to right, through frame scoring, visual memory, audio memory, and using visual memory to allocate the audio budget. Source: paper Figure 2

Step four is the crux of the whole design, and the biggest thing that sets it apart from other compression methods. The paper ran a direct comparison of three "who guides whom" variants.

Even when visual signal is the one doing the guiding, pruning it first before using it to allocate audio beats using the raw visual signal directly by 1.7 points. Raw video carries a lot of repeated background, and using it unfiltered to judge which audio matters lets that noise leak straight into the decision.

Training-free — just drop it in

This part of OMAC doesn't touch the underlying model and needs no retraining; it plugs in as a preprocessing step in front of a model that's already deployed. For teams already running a full omni-modal model in production, that means no rebuilding the training pipeline and no re-running a launch process.

Faster without getting dumber: how accuracy goes up after compression

Compression usually costs accuracy, because the model was originally trained with full material and doesn't adapt well when you suddenly hand it 30%. O-MARC's training stage exists specifically to make up that gap.

During training, the same question produces two answers: the yellow branch on top is the teacher, fed the full token set with no compression; the blue branch below is the student, working from OMAC-compressed material. Each branch computes its own reward, and the gap Δ between them feeds into the compression-aware module in the bottom right, which combines with the reinforcement-learning advantage to produce a new advantage value. Source: paper Figure 3

The key is that the teacher and student are the same model, at the same moment, with the same set of parameters. It answers the same question twice: once with the full material, once with the compressed material. The size of the score gap between the two runs tells you how badly that particular question got hurt by compression.

The mechanics live inside reinforcement learning's "advantage value" — a per-answer score that tells the model how much better this attempt was than others in the same batch, which is what it uses to decide which direction to adjust in. O-MARC multiplies "advantage is positive" by "how much this sample fears compression" to get an extra weight, then adds that back into the advantage value scaled by a tunable strength coefficient. Only samples where both conditions hold get a meaningfully larger weight.

The teacher branch passes its preference to the student by rewriting this advantage value, rather than aligning token-by-token probabilities, pushing the model toward the kind of behavior that stays stable after compression. The entire training run used just 3,000 samples and four GPUs with 96GB of memory each.

The final numbers

The base model is Qwen2.5-Omni-3B, with a 30% retention ratio and 32 frames. The four benchmarks are DailyOmni (everyday-video audio-visual understanding), UGC-AVQA (the new one released here), OmniVideoBench (long-video reasoning), and WorldSense (real-world composite perception). The 45.8 figure is the average across all four; broken out, UGC-AVQA sees the biggest gain while the other three land roughly flat or slightly below the uncompressed baseline. The efficiency figures come from a separate experiment using 64-frame long videos.

The advantage grows the harder you compress. At 70% compression on UGC-AVQA, OMAC still holds 49.2 points while OmniZip drops to 47.1. The less material you keep, the more "keeping the right material" matters.

UGC-AVQA: a benchmark built by deleting the audio track and re-testing

Proving a model genuinely listens requires a test it can't pass without listening. This benchmark's filtering method comes down to a single step.

What's left can't be answered from the picture alone. The full benchmark covers 1,000 user-uploaded short videos and 4,816 question-answer pairs across four categories: audio-visual event progression, scene or time transitions, cross-scene alignment, and fine-grained comparison. Every video has a detailed human-written "omni-modal description" covering people and objects on screen, scene changes, actions, sound events, and temporal relationships; 30% of both the candidate questions and the generated training questions were spot-checked and corrected by trained annotators.

The construction pipeline in four steps: collect 1,000 UGC videos, have humans annotate the four categories of audio-visual questions, strip the audio track and use Gemini 3.1 Flash to filter out the hard samples, and use the remainder as training data. The diagram notes the filtered hard-sample set at 206 videos and 1,648 questions, forming the eval set; the remaining 794 videos and 3,168 questions are used for training. Source: paper Figure 1

What the questions actually look like

Four real examples make it clear.

The original figure for the four questions above, from the paper, with correct answers highlighted in green. Source: paper Figure 5

The third question, about color, looks purely visual — but "first asked" and "later explained" are two points in time that can only be located by listening. This category tests whether the two channels line up; watching alone or listening alone can't answer it.

How did models actually do? Across nine models tested, the top score was Gemini-3.1-pro at 69.1; the largest open-source model, Qwen3-Omni-30B-A3B, scored 65.0; and Qwen2.5-Omni-3B, the base model used in this work, scored only 49.0. Most open-source models' weakest category was cross-scene audio-visual alignment — the same type as the vosotros question above — where even Qwen3-Omni-30B-A3B only managed 60.9.

Data from paper Table 2. Gemini-3.1-pro and flash-lite were the models being scored on the leaderboard; Gemini-3.1-Flash, a different variant, served as the judge that filtered the questions in the first place. Also worth noting: getting a question wrong while silenced shows it isn't solvable from visuals alone, but ambiguous phrasing, poorly designed distractors, or a model's own limitations can also cause a wrong answer — those questions get kept too.

Where that saved third can actually go to work

Cutting both latency and memory by roughly a third means, in practical terms, the same machine can now handle more cameras and more simultaneous live streams.

One more: instructional and training video

The key content in a course video often lives in what's being said, not what's on screen. A slide might just be a static image while the real information comes out of the instructor's mouth. Content understanding, quality assessment, and knowledge extraction for this kind of material require genuine joint audio-visual reasoning — watching alone won't surface anything useful.

The use cases above come from Memories.ai's own research page. The paper itself doesn't include any deployment experiments — every result was measured on Qwen2.5-Omni and Qwen3-Omni benchmarks.

What you can get your hands on right now, and what you still can't

If you want to try this yourself, here's what checking things directly turned up.

One more note: the released dataset ships the original video links and annotations, not the video files themselves. The authors state in the paper's limitations section that some portion of the data will inevitably go dead over time as platform content changes — a common problem for any short-video dataset that depends on external links.

These results suggest that audio-visual compression shouldn't be treated merely as an inference-speedup trick — it's also a trainable condition pointing toward more robust, more efficient omni-modal reasoning. Paper, conclusion section

What this research lands on is turning "smart discarding" from an inference-time trick into a trainable objective. The code is public; the data and weights are still pending.

Memories.ai's approach is "smart discarding": out of the long input string that video and audio get converted into, only 30% is fed to the model — the rest is thrown out as noise. Which 30% gets kept is where all the work is.

The release also comes with a new benchmark, UGC-AVQA, built specifically to fix the hidden problem above: every candidate question has its audio track stripped entirely, and a strong model answers using visuals alone. Anything it gets right is thrown out — only what it can't answer survives. What's left can't be answered straight from the picture.

Keeping just 30% means one wrong cut and the answer is gone. So everything hinges on the order in which things get picked.

Picking happens in four steps: score every frame against the question first and pick the relevant ones, then within those frames keep whatever looks least like the background. On the audio side, keep a few key segments and fold whatever's discarded into the nearest one. Step four is where this method pulls ahead of everyone else — it looks back at where frames were kept densely versus sparsely on the timeline, and uses that density to reallocate the audio budget.

The official numbers are percentages. Converted into what a machine would actually have to do, it's easier to feel: say there are 1,000 roughly one-minute surveillance clips to process.

One thing that's easy to conflate: the training-free plugin used on its own scores 42.8, a bit below the uncompressed 44.1, in exchange for a third off both speed and memory. Getting to that higher 45.8 score requires a follow-up training round — 3,000 questions, four GPUs with 96GB of memory each. Drop it in and use it and drop it in and get more accurate are two different deals.