A Mac Menu Bar Search Tool Built on Google's Multimodal EmbeddingGemma 2: Search Video and Audio Clips Too

DigUp is a Mac menu bar search tool built on Google's multimodal embedding model EmbeddingGemma 2, allowing you to search across selected folders for all kinds of content, including photos, documents, code, as well as video and audio clips. Independent developer Abdur Rahim has open-sourced the app. Describe what you remember to find matching local files, then jump to the relevant video timestamp, document page, or section of code.

The built-in Mac Spotlight already searches filenames, document text, and metadata. DigUp takes this a step further by processing scenes in photos, spoken topics in audio recordings, and visual scenes in videos, such as:

• A video you shot where you only recall seeing a zebra, but cannot remember the filename;
• A podcast episode you listened to where sleep was discussed, but you're left guessing across the scrubber bar;
• A lengthy PDF lease contract where you don't know which page contains the pet clause;
• An exponential backoff retry function in a local repository where neither the filename nor function name contains the word retry.

Powered by Google DeepMind's multimodal embedding model EmbeddingGemma 2 (an embedding model encodes content into comparable sequences of numbers known as vectors), DigUp generates a unified vector representation locally on your Mac across images, screenshots, PDFs, text documents, audio, and video. This enables semantic retrieval based on "what a file contains" rather than "what it is named".

Taking it further, DigUp doesn't just point to the file itself—it jumps directly to the relevant position inside the file: seeking the video player to the timestamp of the scene, navigating a podcast to the audio interval covering the topic, flipping a PDF to the page with the relevant clause, or locating the relevant section of code in your editor. All chunking and retrieval run locally on Apple Silicon without uploading personal files to external services.

Why a Single Sentence Can Find Photos and Audio Recordings

To understand DigUp's retrieval mechanism, the core lies in its underlying multimodal embedding model EmbeddingGemma 2 (open-sourced by Google DeepMind under the Apache 2.0 license).

A common approach to multimodal retrieval relies on intermediate text conversion: using speech recognition to transcribe dialogue, running OCR to extract on-screen text, and then computing similarity over pure text. This pipeline depends heavily on text as a bridge, making it prone to missing non-textual information—such as tone of voice, ambient background sound, visual composition, physical actions, and non-textual visual details.

EmbeddingGemma 2 adopts a Native Multimodality approach:

1. Unified Vector Space: The model has 740M total parameters (comprising a 270M text backbone, a 170M vision encoder, and a 300M audio encoder). Text, still images, sampled video frames, and audio slices are all mapped into the same 768-dimensional vector space.
2. Cross-Modal Comparable Representations: A unified space allows a text query to directly compute cosine similarity against image, audio, or video segments. The application ranks candidate items based on these scores before letting you inspect the original file.
3. Task Instruction Prefixes: The model was trained with task-specific text prefixes to guide different retrieval modes. Standard search queries use task: search result | query: {query}; code retrieval queries use task: code retrieval | query: {query}; document contents are formatted as title: {title} | text: {content}. These prefixes help align text vectors with specific retrieval contexts.
4. Decoupled Modular Encoders: The vision and audio encoders operate independently. During search stages where an input query only needs to be matched against precomputed vectors, only the text backbone needs to be loaded, without keeping vision and audio encoding modules in memory.

After Finding the File, How It Jumps to the Right Page and Timestamp

If an entire video or a multi-hundred-page PDF were encoded into a single vector, retrieval results could at best identify the file as a whole. DigUp's actual implementation splits different media types into segments accompanied by timestamp or location metadata, enabling direct internal navigation.

On macOS, DigUp leverages system frameworks—AVFoundation, PDFKit, Vision, and ImageIO—to handle file parsing and feature extraction:

1. Video Slicing: Keyframe Sampling and Perceptual Hash Deduplication

Video represents the heaviest data footprint. For background processing, DigUp employs a frame budget and deduplication strategy (see source in Media+Apple.swift):

• Sampling Step and Frame Budget: The sampling interval is calculated as max(3, duration / 1200) seconds. Typically, it samples one frame every 3 seconds; as video duration increases, it dynamically widens the step size with a budget cap of 1,200 frames across the video, rather than truncating at the first 1,200 frames. Each frame is encoded with a budget of 70 visual soft tokens;
• Average Hash (aHash) Deduplication: Each extracted frame is downscaled to an 8×8 grayscale image to compute an average hash. If the Hamming distance between consecutive frames is 4 bits or fewer ((hash ^ lastHash).nonzeroBitCount <= 4), the frames are considered visually near-identical. DigUp skips encoding the new frame and simply extends the time interval represented by the prior frame. This deduplicates near-static footage like presentation slides or fixed camera shots, determining which frames represent subsequent intervals;
• Decoupled Visual and Audio Pipelines: Video files are separated into distinct visual and audio paths. Frames are sampled and encoded for visual scenes, while the audio track is extracted and passed to the audio pipeline, independently powering visual action search and spoken dialogue search.

2. Audio Slicing: Sliding Windows and Silence Removal

For audio processing, EmbeddingGemma 2's general model card supports an 8,192-token context window capable of theoretically accommodating longer audio; however, DigUp's current runtime path sets a single-input cap of 30 seconds (source comments note that content beyond 30 seconds is ignored per input):

• Window Specifications: Audio is resampled to 16 kHz mono and segmented into fixed 30-second slices with a 25-second stride (leaving a 5-second overlap between adjacent windows). This 5-second overlap allows some phrases spanning window boundaries to fall within a complete window;
• Silence Removal: The root-mean-square (RMS) energy of each slice is calculated. If RMS ≤ 0.003 (roughly below -50 dBFS), the segment is treated as silence or negligible noise and skipped without generating an embedding, saving storage space.

3. PDFs and Long Documents: Two-Stage Scheduling and Hybrid Text-Image Processing

For lengthy documents and contracts, DigUp uses a batched reading mechanism:

• Page-Count Routing and Two-Stage Scheduling: The source defines bigPDFPages = 50. PDFs with up to 50 pages are indexed entirely during the initial pass; for PDFs exceeding 50 pages, the first pass only reads the first 30 pages, ensuring most documents become searchable quickly, while pages beyond 30 are deferred to background processing (with a per-file ceiling of 5,000 pages). Plain text documents follow a similar strategy, prioritizing the first 40 chunks (~64,000 characters) before indexing remaining text;
• Visual-Text Blending for Scans: When page text extraction yields fewer than 200 characters (common in scanned documents or slides), DigUp renders the page as an image and invokes the vision encoder; simultaneously, it applies Apple Vision OCR, allowing numbers and text on scanned invoices or receipts to be captured by keyword search.

4. Code Repositories: Dedicated Index and Structured Line Chunks

When enabled, code search operates against a dedicated database (code.sqlite):

• Line-Based Chunking: Code is split by full lines, with chunk sizes capped at roughly 1,800 characters. Code comments note that overly short chunks fragment function logic, while overly long chunks dilute retrieval intent;
• Rule Filtering: Respects .gitignore rules, actively skipping dependency packages, generated test artifacts, cryptographic keys, and certificates.

This window- and frame-based chunking mechanism also introduces a granularity trade-off: system navigation points to the matching time window or sampled frame rather than a frame-by-frame or word-by-word micro-anchor; if a fleeting visual moment appears only briefly and falls entirely between sampling intervals, it may be missed.

Balancing Search by Meaning and Search by Exact Words

Once vectors for all segments are stored in the local database, comparing raw cosine similarities between a text query and various modalities (text, images, audio/video) often encounters cross-modal distribution shifts.

1. Modality Centering and Normalized Scoring

In practice, raw cosine similarity distributions for text, images, and audio under the same query can exhibit differing baseline mean levels. If results are ranked directly on raw similarity scores, a modality with a naturally higher mean could dominate top results, pushing down genuinely relevant candidates from other modalities.

DigUp addresses this in Searcher.swift through Modality Centering:

1. Compute Modality Means: Calculates average cosine similarities separately for text, image, and audio candidates under the current query. If a modality contains fewer than 20 vectors, it falls back to the global mean across all vectors for that modality's baseline;
2. Modality Centering: Calculates the centered difference for each candidate segment: centered difference = cosine similarity − modality mean;
3. Joint Standard Deviation and Calibrated Scoring: Computes the mean of centered differences across all calibration candidates, along with a cross-modal joint standard deviation (floored at 0.02), yielding a normalized Z-score: Z = (centered difference − mean of calibration centered differences) / joint standard deviation;
4. Prominence Gate Filtering: Sets a threshold of zGate = 2.0. This score reflects how many standard deviations a segment deviates from its modality average—evaluating relative prominence within its modality rather than representing an absolute probability or retrieval precision metric. Using a shared cross-modal joint standard deviation prevents modalities with narrower variance from being artificially amplified.

2. Exact Match Complement: Why FTS5 Full-Text Search Is Needed

Embedding models excel at grasping natural language intent, but vector representations cannot reliably preserve character-exact identifiers, serial numbers, or invoice IDs.

Accordingly, DigUp employs a dual-track hybrid retrieval architecture:

• Semantic Track (Meaning): Powered by EmbeddingGemma 2 vector similarities and modality-centered scores;
• Exact Track (Words): Powered by SQLite's native FTS5 full-text search module, performing stemming and normalized indexing over filenames, directory paths, document text, and OCR-extracted characters.

The system dynamically balances weights across both tracks based on query characteristics:

• General Search: Guided primarily by semantic vector scores, with keyword hits offering additive bonus score;
• Lookup for Identifiers: In source code, the isLookup check requires that all effective terms in the query contain digits, and either contain letters or have a digit length of 4 or more. Entering a standalone identifier like 4815162342 or a code like K7Q2LM triggers this mode, doubling the keyword weight while halving the semantic vector weight; however, queries mixing standard vocabulary (such as invoice 4815162342, where invoice is not a stopword and contains no digits) will not trigger this special weighting;
• Tuning for Code Retrieval: In code search, frequent words like app, get, or running appear extensively throughout source files; thus, the keyword weight in code search is lowered to 1.0 (compared to 2.0 in general search) to prevent superficial token frequency from distorting true intent.

Switch through the query examples below to see which index they route to and how meaning and exact words participate in retrieval:

What the Four Search Modes Return in the Author's Demo

In the walkthrough published by developer Abdur Rahim (and reshared by the official Google Gemma account), several typical interaction workflows are highlighted:

Case 1: Video Scene Retrieval (zebra in a video)

Typing zebra in a video in the search bar matches a video named Masai Mara - day 2.mp4 with an associated timestamp.

• Direct Action: Pressing Enter opens QuickTime Player and seeks straight to the relevant playback position featuring zebras;
• Permission Model: Under macOS automation control permissions, the first time DigUp controls QuickTime Player, a system confirmation dialog appears; once granted, the app can seek the player directly.

Case 2: Podcast Discussion Retrieval (where they talk about sleep)

Typing where they talk about sleep matches the audio file The Long Run - Ep 14.mp3.

Case 3: Long PDF Clause Retrieval (the clause about pets in the lease)

Searching for pet restrictions in a long-term lease agreement locates page 4 of Lease - 14 Elm Street.pdf.

Case 4: Code Intent Retrieval (code: retry with backoff)

When a query begins with the code: prefix, DigUp searches configured code repositories.

In code retrieval logic, intent-based searches return relevant stretches of lines; only when entering an exact symbol or identifier does the system attempt to pinpoint a specific definition line. In addition to Zed, DigUp supports editors including VS Code, Cursor, Xcode, Sublime Text, and JetBrains IDEs.

Multilingual Capabilities and Chinese Support

• Cross-Lingual Semantic Capability: While EmbeddingGemma 2 was pre-trained across 100+ languages, performance is not uniformly consistent across all languages; the project's own evaluation benchmark currently focuses on English, Bengali, and Arabic. The Chinese query examples provided in this text serve as editorial translations for Chinese readers, not project evaluation cases;
• Keywords and OCR Recognition: In FTS5 full-text indexing, SQLite's default tokenizer does not segment words for non-space-delimited languages such as Chinese, Japanese, or Thai; for text recognition in images and screenshots, capabilities depend on the Apple Vision framework (which supports around 25 languages);
• Input Method Experience Improvements: In version 0.6.1, the developer resolved an issue where composite input methods (IMEs) for languages like Chinese, Japanese, and Korean broke up uncommitted phonetic keystrokes in the search bar, ensuring continuous typing input.

Keeping Data Local: How Compute and Storage Are Structured

In local resource utilization and system architecture, DigUp operates without depending on external daemon processes or standalone vector database services:

1. Single-File Local Storage and Vector Footprint: DigUp stores file records, chunk metadata, and 768-dimensional float16 vectors within a single local SQLite database file. Taking 20,000 vector segments as an example, raw vector data occupies roughly 30MB (reflecting raw vector storage alone, excluding associated metadata and application runtime overhead). During search, these vectors are loaded into memory for direct dot-product matrix operations, eliminating the need to maintain an external vector database service.
2. Helper Process Exits Upon Completion: Resource-intensive file scanning and GPU vector encoding are delegated to an independent Helper process, leveraging llama.cpp's Metal backend to tap Apple Silicon GPUs. Once the pending processing queue empties, this helper process exits and releases its compute resources; when a user initiates a search, the main menu bar application loads only the text query encoder on demand (README estimates ~250MB memory), otherwise remaining lightweight in the menu bar.
3. Scans Only Selected Folders: DigUp reads files strictly within user-designated directories, never modifying, moving, or uploading indexed content. During scanning, it automatically bypasses credentials, certificates, password files, hidden folders (such as ~/.ssh), application bundles, caches, and dataset-like folders; cloud placeholder files stored in iCloud that have not been downloaded locally will not trigger automatic downloads. Network activity is limited strictly to initial model weight downloads (~865MB) and daily update checks (configurable in Settings), with no user accounts or telemetry tracking. The application is open-sourced under the MIT license, with the underlying model licensed under Apache 2.0; Windows and Linux versions are currently planned.

Walkthrough on Your Own Mac

Users with an Apple Silicon Mac (M-series chip running macOS 14 or later) can get started through the following steps:

Step 1: Download and Installation

• Install via Homebrew Cask (requires Homebrew installed, run in Terminal):

  brew install --cask arahim3/tap/digup
  

• Manual DMG Download: Head to the project's Releases page to download signed DigUp-0.6.1.dmg, open the DMG, and drag DigUp into your Applications folder.

Step 2: Model Download and Local Import

1. Once launched, DigUp resides in your menu bar;
2. On first launch, it automatically downloads EmbeddingGemma 2 model weights (~865MB total: ~310MB for the text backbone and ~555MB for audio/vision encoders);

   Manual Model Import: If direct downloads are blocked on your network, go to Settings → Storage and click Use Files… to point to model files manually. The required files are Q8_0 quantized weights maintained by ggml-org: embeddinggemma-2-Q8_0.gguf and mmproj-embeddinggemma-2-Q8_0.gguf. You can download both files from Hugging Face at ggml-org/embeddinggemma-2-GGUF (pinned commit bfcd298762cc34d0357ece5ebdd31791a3a374d8). Once selected, DigUp verifies and loads them.

Step 3: Configure Folders and Indexing Rules

Open DigUp's Settings window to select folders for indexing:

• Getting Started Recommendation: Begin by selecting a smaller local directory with a limited number of files, monitor the menu bar progress indicator, and try searching once indexing finishes;
• Time Expectations: Indexing several hundred standard files typically takes just a few minutes; for large folders like 'Downloads', the README notes initial scanning can take roughly an hour;
• Low Power Mode and AC Power Options: Background indexing automatically pauses when macOS enters Low Power Mode and resumes once disabled; requiring AC power for indexing is an independent toggle;
• Scheduled Indexing: To avoid daytime system load, enable scheduled indexing in Settings → Folders (configured by default between 1:00 AM and 7:00 AM).

Step 4: Common Global Shortcuts

Step 5: Copy a Query and Inspect Matching Source Files

When searching, express your intent in a single language (English or Chinese), provided your chosen index directory contains matching content.

The following English queries are from official project examples, alongside Chinese reference translations. Pick either to test:

Video Scene

zebra in a video

Chinese reference example (editorial translation):

有斑马出现的视频

Podcast Topic

where they talk about sleep

Chinese reference example (editorial translation):

讨论睡眠的那段录音

PDF Clause

the clause about pets in the lease

Chinese reference example (editorial translation):

租约中关于宠物的条款

Code Intent

code: retry a failed call with backoff

Chinese reference example (editorial translation):

code: 调用失败后按退避策略重试

Enable Code Search First: Code search is disabled by default. To search code, first navigate to Settings → Code, add your repository directories, toggle the feature on, and select your preferred editor (such as Zed, VS Code, Cursor, Xcode, etc.). Then start queries with the code: prefix. We recommend checking the line preview in the right panel before pressing Enter; if results do not appear as expected, verify that the directory has finished indexing, check that files are not cloud-only placeholders, and refine your descriptive query phrasing.

When Filenames Fade, Content Remains the Entry Point

DigUp expands the entry points for navigating local files: enabling scenes in photos, video shots, long audio recordings, complex document clauses, and code logic to be retrieved and targeted through natural language. Running entirely on local silicon without uploading private data, it offers a lightweight and practical path for multimedia asset retrieval on the Mac.