Repository navigation
EmbeddingGemma 2 audio: index an hour of audio in ~5 s, search it with text - #31
Merged
Merged
Conversation
…th text EmbeddingGemma2Audio runs the model's audio encoder (Gemma 4 USM conformer, 12 layers, log-mel computed in-graph) from the new EmbeddingGemma2Audio.mlpackage on FluidInference/embeddinggemma-2-coreml (revision c0fefd0a, pinned + checksummed; the text bundle keeps its own revision and compile stamp, so nothing recompiles). Each 10 s window of 16 kHz audio becomes 250 tokens, which the text model embeds as <bos> <|audio> tokens <audio|> <eos> through embed_256 — audio lands in the text space and is searchable with a text query. - Audio encoder on the GPU: the chunked local attention's 5-D blocks fall back to the CPU on the ANE (46 ms per window vs 13.6 ms on the GPU). Windows run four in flight across all files, so the GPU (audio) and the Neural Engine (text) overlap: 1 hour in 4.6-5.7 s (626-777x real time), 4 h 36 min across 2,301 files in 34.7 s (478x). - fp16 drifts on a few quiet tokens; on a 1-hour index the top hit matched fp32 for 15/15 queries (top-5 overlap 95%). Swift end to end vs sentence-transformers fp32: cos 0.9995 on a LibriSpeech clip. - Any AVFoundation format decoded to 16 kHz mono; window/frame arithmetic matches the HF feature extractor (999 frames / 250 tokens per 10 s, 347 / 87 for 3.48 s), covered by unit tests. AudioSearchCheck: index files or folders, timing, --query, --dump. AudioSearchDemo: index the public datasets on this Mac (Earnings-22 hour, FLEURS en/fr, ESC-50) or --audio= folders, live speed tiles, search as you type (one sgemv over all windows), play the matching 10 s; demo.sh adds macmon + the live log. Speech queries work well; sound events are weak in the model itself (ESC-50 zero-shot 24% over 50 classes, same as PyTorch fp32).
…uery, play its top 3 Stop/Auto (Space) toggles it; typing a query or pressing play ends it; --manual starts without it, --clip= sets seconds per result. Search now ranks through one shared path (rank) for typed and hands-free queries.
… s of search Hands-free now alternates two segments (--segment=, default 30 s): - Speed: read and embed every file again as fast as it can, with live speed tiles. The first pass runs to completion and is searchable as it grows (windows stream in through EmbeddingGemma2Audio's new onWindow callback); later passes build a new index beside the current one and are cut at the segment end, so searches never see a partial index. - Search: type each suggested query and play its top three (4 s each, --clip=), continuing through the suggestion list across segments. Fixes found on the way: the field's change callback echoing a query the show typed (or cleared) switched hands-free off; the speed countdown returned before the indexing task had started, so a timed pass ran to the end. EmbeddingGemma2Audio checks for cancellation after each window.
… loop After the first read, the show alternates 30 s of listening (type a query, play its top three) with 30 s of search speed: ~100 different queries (suggestions, earnings/news topics, the ESC-50 sound classes) back to back, 64 at a time, embedded eight per Neural Engine call (pack_256) and ranked against every window with one sgemm; each query gets its own top 10, the latest is shown. Tiles switch to searches run, searches/s (rolling 1 s), ms per search and windows ranked per second. ~700 searches/s, 1.4 ms each (embed 1.4 ms, rank 0.02 ms) over 2,786 windows. The packed path is warmed at load so the first second is not a cold start. The timed re-read segments are gone.
Indexing starts at launch with its own tile row; the listen / search-speed show starts as soon as 64 windows are searchable and searches whatever is indexed by then. Search speed waits for indexing to finish (both use the Neural Engine; during indexing it measured 75/s instead of ~700/s), so the first listening segment runs until then. Search tiles get their own row.
The show waits until every file is indexed (status: indexing · search starts when done), then alternates listen and search speed.
…n stop Ends with a summary line (audio indexed and its speed, searches run at their average rate); Auto (Space) runs it again.
Three buttons replace Index / Auto / Reset. Play (↩) runs the show once: index every file, 30 s of listening (type each suggested query, play its top three), 30 s of search speed, done with a summary. Pause (Space) holds it anywhere: indexing (EmbeddingGemma2Audio.embed gains a beforeWindow hook used as a pause gate), typing, the playing clip, the speed loop, and the step timers and speeds, which count running time only. Replay (⌘R) clears the index and runs it again. Nothing starts on launch (--autostart presses Play); the search field, suggestions and row play buttons work when the show is not running.
forward-looking statements disclaimer, a baby crying, the operator opens the line for questions, someone laughing (two earnings-call phrases, two sounds; all four return three right hits). Clips are 3 s; the step is no longer cut at 30 s.
…e names The listening step plays the top 3 of eight searches: two earnings-call phrases (forward-looking statements, the operator opening the line) and six sounds (baby crying, church bells, siren, laughing, cow, birds). With the ESC-50 clips renamed to random ids (clip_3f9a1c.wav; classes kept in esc50/labels.json, outside the app's view) all 24 hits are right, so the file name plays no part; the header says so. Search-speed queries for the sound classes come from labels.json. Search speed stays 30 s (21,312 searches at 710/s over 2,786 windows).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds EmbeddingGemma 2's audio encoder (Core ML, from FluidInference/embeddinggemma-2-coreml @ c0fefd0a) and a search-audio demo.
Quality note: speech queries work well; sound-event search is weak in the model itself (ESC-50 zero-shot 24%, identical in PyTorch).
Checked locally: release build, swift-format lint, AudioSearchCheck on the Earnings-22 hour / ESC-50 / LibriSpeech; unit tests for the window arithmetic run in CI.
🤖 Generated with Claude Code