Google Open-Sources Multimodal Embedding Model EmbeddingGemma 2: Search Text, Images, Audio, and Video with Natural Language

Phones store photo albums, meeting recordings, and videos, but when searching for a specific piece of content, you rarely remember the file name. Searching by content requires a distinct class of models—embedding models(algorithms responsible for compressing real-world content such as text, images, and audio into high-dimensional mathematical vectors, enabling computers to compare semantic similarity). Since its release, the previous-generation text-only compact model has accumulated over 20 million downloads, handling text materials; the new generation expands further to images, recordings, and videos.

Google DeepMind has introduced EmbeddingGemma 2, a brand-new open-source multimodal embedding model built for on-device deployment. Built on the Gemma 4 architecture under a commercially permissive Apache 2.0 license, it expands context length fourfold to 8,192 tokens and, at 740 million (740M) total parameters, brings text, code, images, video, and audio all into a single 768-dimensional vector space. With a detachable modular design and Matryoshka-style vector truncation, it allows loading only the necessary encoders for a given task and lets developers trade shorter vectors for lower storage overhead.

How a Single Search Query Finds Text, Images, and Audio

A common approach in local search or private RAG(Retrieval-Augmented Generation, where relevant materials are retrieved first and then passed to a generative model to answer) systems is to convert different formats into text first: to search for a video in a photo album, the system must call a speech recognition model to transcribe dialogue in the video, call an image recognition model to generate descriptive tags, and finally compare them using a text embedding model. This adds processing stages and can lose information that is difficult to describe in short text—such as the background sound of ocean waves, guitar chords, or the visual atmosphere of a specific camera movement in a video.

The fundamental breakthrough of EmbeddingGemma 2 lies in native multimodal alignment. It ensures text, images, video, and audio share the exact same 768-dimensional mathematical space right from training. For instance, in the official demo video, a photo of a cat, the text "cat", and the sound of a cat meowing are all mapped to nearby positions in this shared high-dimensional space after encoding; precisely because content across different modalities is projected into the same vector space, the system can compare their relevance directly by computing geometric distance or cosine similarity, without first translating audio or video frames into text.

Even when queries and stored materials differ in format, semantic similarity can be compared directly. In practical implementations, this process follows a clear step-by-step mechanism:

1. Pre-encoding materials and indexing: Photos, video clips, or audio recordings in local photo albums are encoded in advance via EmbeddingGemma 2 into 768-dimensional vectors and stored in an on-device vector index.
2. Real-time query encoding: The system encodes incoming queries into vectors in the same space. For example, if a user speaks "find videos of the sunset by the sea", this voice recording can be directly converted into a vector and compared with vectors of video clips. This serves as an illustrative example to explain the workflow.
3. Same-space similarity matching and retrieval: The similarity between the query vector and each candidate vector is computed within the unified vector space, ranked by score, and candidate original items are retrieved.
4. Passing to an LLM only if generation is needed: If the goal is purely retrieval and pinpointing, candidate originals are presented directly to the user; if the application requires further summarization, Q&A, or reasoning, the retrieved candidate content is passed to an on-device generative model such as Gemma 4 to produce an answer.

The same representations can also be used for classification and clustering. Classification compares input vectors against predefined category vectors to pick the closest match; Google's MediaPipe Decision Task API mini-game demo selects among predefined actions like JUMP, DUCK, and SPRINT based on multimodal inputs.(representing jump, duck, and sprint respectively) Clustering, by contrast, does not specify categories in advance, but instead groups items with nearby vector positions, making it easy to organize content across diverse formats.

EmbeddingGemma 2 outputs numerical representations. Whether retrieved content can answer a question still requires subsequent processing; when these steps run on-device, vector extraction and comparison are executed entirely on local chips, keeping photo albums and audio recordings from ever uploading to cloud servers, reducing network round-trips while keeping original data on the device.

Why You Don't Need to Load the Entire Model for Text-Only Search

Applications that only need to search text or code do not need to load image and audio components into memory. EmbeddingGemma 2 adopts a modular architecture:

The overall model consists of 24 Transformer layers with an internal hidden dimension of 512, which are then projected to standard 768-dimensional vectors. Its 740 million (740M) total parameters can be selectively loaded depending on the task:

1. Text backbone (270M parameters): Comprising a 130M-parameter Transformer trunk and a 140M-parameter embedding layer, handling the parsing of all multilingual text and code.
2. Vision encoder (170M parameters): Dedicated to parsing static images and sampled video frames.
3. Audio encoder (300M parameters): Dedicated to parsing multilingual speech and various environmental sounds.

Which components to load depends on what materials the application handles:

• Text/code-only tasks (Text only): By configuring the vision and audio encoders as null in frameworks like Sentence-Transformers, the model loads only 270M parameters. In Google's quantized benchmarks on Pixel 11 Pro, active RAM usage for text weights drops as low as approximately 191 MB.
• Text + Vision retrieval tasks: Disabling audio components brings the combined parameter count down to just 440M.
• Text + Audio tasks: Disabling the vision encoder results in 570M combined parameters; in scenarios that only involve speech or environmental sounds without visual retrieval, the vision encoder can be left unloaded.
• Full Multimodal: Mounting all components reaches 740M parameters. When quantized on the same phone, active RAM usage for full multimodal weights is as low as approximately 567 MB. Neither figure includes the overhead of the broader application, retrieval index, or generative model.

Furthermore, because EmbeddingGemma 2 derives from the Gemma 4 architecture, it shares the same text tokenizer and audio encoder with Gemma 4 generative models. This means that if developers build a complete on-device RAG pipeline on mobile (with EmbeddingGemma 2 responsible for finding materials and Gemma 4 for summarizing answers), both models can share common underlying weights in memory, helping reduce joint runtime memory footprint.

8K Is a Shared Budget: Mixed Inputs Must Be Counted Together

The previous-generation text-only model had a context window of only 2,048 (2K) tokens, often forcing lengthy technical documents or multi-turn dialogues to be chopped into tiny chunks. EmbeddingGemma 2 extends context fourfold to 8,192 (8K) tokens.

Importantly, these 8,192 tokens form a unified measurement pool shared across all modalities. The model defines a clear conversion standard for each modality:

• Text and code: Following standard tokenization, approximately 1 subword(the smallest semantic unit split by a tokenizer) consumes 1 token. For text-only input, this allows up to 8,192 tokens.
• Audio signals: Requiring 16 kHz mono audio, audio consumes tokens at a fixed rate of 25 tokens per second. In audio-only mode, the 8K window is sufficient to directly fit continuous audio up to 327 seconds (approximately 5.5 minutes).
• Video signals: Operating by default at 1 fps(extracting 1 frame per second), each video frame consumes 140 tokens by default, allowing a maximum of approximately 58 frames per input (i.e., around 58 seconds of sampled video).
• Static images: A single image consumes 280 tokens by default, accommodating up to approximately 29 images in a single pass.

These ceilings are calculated individually for each single modality. When text, images, video, and audio are mixed in an input, they consume this 8K budget together and cannot max out their individual limits simultaneously. This also clarifies the relationship between searching through a long audio library and the 8K single-pass encoding limit: to realize the stated capability of "finding content in hours of recordings," long recordings can be segmented into context-sized chunks, indexed separately, and retrieved according to the query. That is fundamentally different from encoding hours of audio in a single pass.

More flexibly, Google also provides an elastic visual soft token budget adjustment mechanism: developers can freely tune the token allocation per image or video frame between 70 and 1,120 soft tokens based on task requirements. For general classification where fine detail is not critical, dialing down to 70 tokens allows packing up to around 114 images or video frames into a single input; for visual documents like detailed circuit schematics or complex engineering charts, developers can increase the token quota, trading slightly more compute for finer visual representations.

Furthermore, EmbeddingGemma 2 supports genuine freely interleaved multimodal input. Within input text, you can place <|image|>, <|video|>, and <|audio|> placeholders at any point. For example, when indexing a trail running shoe for sale, the input can be written as:

emb = model.encode({
    "text": "越野防水跑鞋。<|image|> 采用透气网面设计。<|image|> 湿滑岩石抓地力实测：<|video|>",
    "image": ["shoe.jpg", "mesh.jpg"],
    "video": "demo.mp4",
})

Based on the relative positions of the placeholders, the model fuses the text description, two close-up detail photos, and a short test video into a unified 768-dimensional vector. A user searching with a simple natural language query like "shoes with grip on wet rock" can subsequently retrieve this comprehensive multimodal record.

Shortening Vectors: How Much Storage Is Saved and How Much Quality Is Lost

On-device platforms face not only constrained compute cores, but also precious local storage and memory. Storing millions of traditional 768-dimensional float vectors imposes a substantial burden on mobile storage.

To address this, EmbeddingGemma 2 natively integrates MRL(Matryoshka Representation Learning). By optimizing loss during training, this technique ensures multiple predefined prefix dimensions serve as valid, standalone representations, though it does not guarantee every individual dimension is strictly sorted by importance. Consequently, developers can truncate 768 dimensions down to 512, 256, or even 128 dimensions simply by slicing prefix dimensions at export time, without retraining:

Official benchmark data yields clear engineering selection heuristics:

1. 256 dimensions serves as a practical starting point for evaluation: Compared with the full 768 dimensions, 256 dimensions reduces vector storage footprint to one-third, with minor quality degradation on official benchmarks (multilingual only drops from 61.36 to 60.41, code from 78.68 to 76.18, and audio decreases by roughly 0.48 points), though whether it meets application requirements still depends on retrieval results on actual data.
2. Exercise caution with 128 dimensions in multimodal settings: While 128 dimensions shrinks vector footprint to one-sixth and maintains decent metrics on text and code, information loss is pronounced in complex vision and cross-modal benchmarks (MMEB v2 plunges steeply from 59.01 to 45.65). Therefore, 128 dimensions is recommended only for ultra-constrained text-only on-device scenarios; using it in multimodal applications requires rigorous offline domain validation.

Evaluating Compact Models: Measuring Quality Alongside Scale

Evaluations published by Google show that it approaches or surpasses several larger models at a smaller scale, although larger models in the chart still achieve higher scores. The value here lies in quality under constrained resources, rather than ranking first on every metric:

• Improved code retrieval: On the full-precision MTEB Code (v1) benchmark published by Google, EmbeddingGemma 2 achieved a score of 78.68(NDCG@10, measuring ranking relevance across top-10 retrieved results), representing an increase of 9.92 points (a relative improvement of nearly 14%) compared with 68.76 from the first-generation text-only model. These results support its use for local IDE code indexing, semantic code search, and context recall for on-device coding Agents. In the comparison charts accompanying the blog post, the 270M text core scored higher on code retrieval than some smaller models (such as Qwen3 0.6B at roughly 75.4) and approached 8B-scale models several times its size (around 80.8).(Note: Visual readings from plotted points on the official chart indicate approximately 76.7 points, differing from the 78.68 formally listed in the official model card table; exact benchmark scores should follow the numbers in the model card)

• Strong visual and document understanding: Scoring 64.64 on the MIEB Lite image embedding benchmark, it outperforms jina-v5-omni-small (around 61.5) and approaches the 3B-parameter LCO-Embedding-Omni-3B (around 65.7); on multimodal visual document understanding (MMEB v2 VisDoc), it achieved a score of 67.84, though this benchmark measures visual document retrieval and should not be equated directly with full-report reasoning or Q&A capabilities.

• Audio performance surpasses several specialized models in the chart: In MAEB audio benchmark evaluations, the model achieved 49.39 points, and 69.54 points on MSEB retrieval benchmarks. This nears the 3-billion-parameter e5-omni-3B (around 48.6) and exceeds several dedicated audio models shown in the chart (such as large_clap at roughly 34.5, MuQ-MuLan at roughly 28.4, and Qwen2-Audio-7B at around 35).(Note: The original chart's horizontal axis plots the multimodal point at roughly 670M visually, and the scatter plot visual reading shows around 48.7, differing slightly from the model card exact value of 49.39; the model card lists 740M parameters and a score of 49.39)

• Multilingual text on par with predecessor: On the MTEB multilingual (v2) benchmark covering over 100 languages, it scored 61.36, virtually on par with its predecessor (61.15); the primary gains come from code and newly added multimodal capabilities.

Before Integration: Check Numerical Precision, Normalization, and Task Prefixes

When integrating EmbeddingGemma 2 into production pipelines, three configuration details require attention:

Pitfall 1: Strictly avoid float16; bfloat16 or float32 is mandatory

Internal activation ranges in EmbeddingGemma 2 exceed the dynamic representation range of float16. If torch.float16 is used for inference, the model may return NaN(Not a Number) or degraded vectors without raising an explicit error.

• Recommended practice: In hardware-accelerated environments, torch.bfloat16(which shares the same exponent range as float32 while requiring about half the storage per number) must be chosen; in environments lacking native bfloat16 support(the model card recommends float32 for most CPUs), fallback to torch.float32 is required.

Pitfall 2: Re-apply L2 normalization after MRL truncation

Using the dot product of normalized vectors to compute cosine similarity requires vectors to have unit length (norm of 1). When slicing 768 dimensions down to 256, the sum of squares of the first 256 components obviously no longer equals 1. If you forget to apply L2 normalization(a geometric operation that divides a vector by its Euclidean norm to reset its length to 1) after truncation, subsequent vector dot-product comparisons will not throw errors, but can introduce silent bias—yielding similarity scores that appear plausible on the surface but are distorted, silently degrading ranking quality. The official model card recommends handling truncation and normalization directly via call arguments to prevent issues caused by manual slicing.

• Recommended practice: Set automatic normalization parameters directly in library calls, or apply normalization manually:

query_emb = model.encode(
    query,
    truncate_dim=256,          # 声明截断维度
    normalize_embeddings=True  # 强制要求截断后重新归一化！
)

At the same time, note: query and index dimensions must match exactly. Never use a 768-dimensional query vector to search a 256-dimensional vector index.

Pitfall 3: Follow standardized task prefixes and never add prefixes to non-text inputs

EmbeddingGemma 2 was trained with concise task prefixes. Prepending prefixes matching the task before input text helps boost retrieval quality.

• Asymmetric retrieval (e.g., document retrieval, code retrieval, Q&A): Queries and target materials have different structures, requiring distinct query and document prefixes.
  • Web/document retrieval: Query format is task: search result | query: {query}; target documents must strictly prepend titles: title: {title} | text: {content}(if untitled, use title: none | text: {content}).
  • Code search: Queries can use prompt_name="CodeRetrieval", corresponding to task: code retrieval | query: {query}; code files are formatted as title: {file or class name} | text: {code}.
• Symmetric comparison (e.g., sentence similarity, text classification, clustering): Both sides use identical instruction prefixes (for instance, task: sentence similarity | query: {content}).
• Most critical rule: All prefix instructions apply only to pure text! Pass raw signals directly for images, video, and audio—never attach any text instruction prefixes.

From Demo Apps to Developer Tools: How to Get Started

Explore the concrete on-device utility of multimodal embeddings through the official reference applications first:

• Google AI Edge Gallery (Photo and video pinpointing): Includes Instant Media Search for querying albums via text or images, and Video Moments Finder for pinpointing exact moments in videos using natural language or voice. In the official release video, presenters searched a birthday video for "blowing out candles"; the returned result jumped straight to the exact timestamp of blowing out candles on the video timeline rather than just returning the whole video file, allowing users to tap and play immediately from that moment.
• Google AI Edge Foresight (Chaining on-device multimodal RAG): In the demonstration, the application listens to questions in real time in the background, encodes the audio stream, retrieves local text, photos, and system architecture diagrams, and lets an on-device Gemma 4 generate summary answers—connecting "embedding models for retrieval" and "generative models for answering" while keeping sensitive data on the device.
• MediaPipe Decision Task API (Predefined classification and decision routing): Leverages similarity matching between input representations and predefined candidate vectors to provide low-latency, real-time intent classification and decision routing for applications.

Model weights are now officially released on Hugging Face and Kaggle, with forthcoming support on Gemini Enterprise Agent Platform's Model Garden.

Developers can choose entry points based on their runtime environments:

• Model downloads and on-device execution: Weights are available on Hugging Face and Kaggle, and LiteRT Community provides dedicated versions optimized with on-device quantization. Mobile developers can achieve out-of-the-box vector extraction using the MediaPipe suite provided by Google AI Edge, or use LiteRT(formerly TensorFlow Lite) for highly customized model embedding; web applications support fully local, front-end execution via transformers.js and WebGPU.
• Server-side and local toolchains: Tools listed in the release post include sentence-transformers, transformers, vLLM, SGLang, Apple ecosystem's MLX, and developer-oriented tools like llama.cpp, Ollama, and LMStudio; generated vectors can be stored in modern vector databases such as Qdrant; and fine-tuning can follow specialized guidance provided by Unsloth.

In the full Google release video, the team also explained that developers can fine-tune the model to adapt to specialized domain vocabulary and materials (such as legal contracts, medical imaging, or technical product catalogs mentioned in the video). Fine-tuning helps the model better capture vertical-domain representations, improving retrieval precision without expanding model size; however, the legal and medical scenarios mentioned in the video serve only as directional examples, and retrieval efficacy must still be evaluated on one's own data. Developers can consult Unsloth's fine-tuning guides or get started with relevant examples in the Gemma Cookbook.

The model supports over 100 languages, though performance varies across different languages; retrieval quality also depends on actual data and task requirements. EmbeddingGemma 2 generates representations, but applications must still handle post-retrieval filtering and business logic, verifying performance against their own domain data.