From 3cbd5d1f00c34cd7ef34bee83b5b872a357c0a5c Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 11:56:01 +0100 Subject: [PATCH 1/8] fix(transcription): name the model that does not label speakers A provider can offer models with and without speaker labels, so the notice names the model, as the Speaker labels row already does. --- .claude/skills/verify/features/transcripts.md | 2 +- src/services/TranscriptionService.test.ts | 2 +- src/services/transcription/transcribe.ts | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/skills/verify/features/transcripts.md b/.claude/skills/verify/features/transcripts.md index 28512591..a6e1fddc 100644 --- a/.claude/skills/verify/features/transcripts.md +++ b/.claude/skills/verify/features/transcripts.md @@ -57,7 +57,7 @@ Preconditions: - In parts. Write a 70 min MP3 of two voices, a 300 Hz tone and a 700 Hz tone that take turns every 45 s: `ffmpeg -f lavfi -i "aevalsrc='0.5*sin(2*PI*if(lt(mod(t\,90)\,45)\,300\,700)*t)':s=8000:d=4200" -ac 1 -ar 8000 -b:a 8k "$VAULT/Fixtures/Two Voices.mp3"`. Play it and run the audio command on Grok. `$NET log` shows three `POST https://openrouter.ai/api/v1/audio/transcriptions` entries of about 1.9 MB each, one after the other, then the speaker reel, one of about 82 KB whose `decodedSeconds` is `61.56`: six 9 s clips with 1.5 s of silence between two. The fixture numbers the voices in the order they sound in each request, so the second part, which starts with the high tone, numbers them the other way. The note still numbers the low voice `1` and the high voice `2` throughout, the paragraph at `00:23:15` runs on across the first part boundary, and the progress notice ends with `Transcription completed and saved.` With `ElevenLabs Scribe v2` the parts go as multipart, no reel follows, each part's fixed answer numbers two more speakers, and the notice ends with `OpenRouter transcribed this episode in 3 parts, and speaker labels restart in each, so one person can have several labels.` - **OpenAI.** Choose `OpenAI` with its fake key. `Untranscribed Episode` downloads its WAV and `$NET log` shows a `POST https://api.openai.com/v1/audio/transcriptions` answered by the fixture provider. The note holds the fixture text with no timestamps. - In parts. Write an MP3 longer than 2,070 s into the vault, for example 70 min in 4 MB: `ffmpeg -f lavfi -i sine=frequency=440:duration=4200 -ac 1 -ar 8000 -b:a 8k "$VAULT/Fixtures/Long Fixture.mp3"`. Play it ([local-files.md](./local-files.md)) and run the audio command. `$NET log` shows three `POST https://api.openai.com/v1/audio/transcriptions` entries of about 1.4 MB each, one after the other, and the note holds the fixture text three times with no timestamps. - - Speaker labels. With `Speaker labels` on, as in a library that labelled speakers with OpenAI on 2.24.0, the progress notice ends with `Saved without speaker labels, because OpenAI does not label speakers. These providers do: ElevenLabs, Deepgram, OpenRouter.` and stays until clicked. + - Speaker labels. With `Speaker labels` on, as in a library that labelled speakers with OpenAI on 2.24.0, the progress notice ends with `Saved without speaker labels, because GPT Transcribe does not label speakers. These providers do: ElevenLabs, Deepgram, OpenRouter.` and stays until clicked. - **A 2.24.0 library.** Write a 2.24.0 `data.json` state into the vault (`transcript.diarization` with `enabled` and `provider`, no `transcript.provider`), run `CLI --reload`, and read `app.plugins.plugins.podnotes.settings.transcript.provider`: `deepgram` for speaker labels with Deepgram, `openai` for speaker labels with OpenAI or a linked OpenAI key, `elevenlabs` otherwise. `data.json` is unchanged until the next save, which adds `transcript.provider`, `transcript.model`, `elevenLabsSecretId` and `openRouterSecretId` and leaves `transcript.diarization.provider` as it was. - **Proof.** Screenshot the opened note (`npm run capture:e2e-obsidian -- screenshot .obsidian-e2e-artifacts/transcripts/note.png`), keep the note text, `$NET log` and `fixtures.mjs requests` for every run, and read `data.json` to show no new field. diff --git a/src/services/TranscriptionService.test.ts b/src/services/TranscriptionService.test.ts index 468de16e..d64d82ac 100644 --- a/src/services/TranscriptionService.test.ts +++ b/src/services/TranscriptionService.test.ts @@ -486,7 +486,7 @@ describe("TranscriptionService", () => { await vi.advanceTimersByTimeAsync(60 * 60_000); expect(String(setMessage.mock.calls.at(-1)?.[0])).toMatch( - /\n\nSaved without speaker labels, because OpenAI does not label speakers\. These providers do: ElevenLabs, Deepgram, OpenRouter\.$/, + /\n\nSaved without speaker labels, because GPT Transcribe does not label speakers\. These providers do: ElevenLabs, Deepgram, OpenRouter\.$/, ); expect(hide).not.toHaveBeenCalled(); expect(savedNote(plugin)).toBe("# Test Episode\n\nHello there. Hi!"); diff --git a/src/services/transcription/transcribe.ts b/src/services/transcription/transcribe.ts index 6ecab8d8..7e38dbaf 100644 --- a/src/services/transcription/transcribe.ts +++ b/src/services/transcription/transcribe.ts @@ -129,7 +129,7 @@ export async function transcribe(job: TranscriptionJob): Promise { } if (choice.speakers && !model.speakers) { warnings.push( - `Saved without speaker labels, because ${choice.label} does not label speakers. These providers do: ${providersWithSpeakers().join(", ")}.`, + `Saved without speaker labels, because ${model.label} does not label speakers. These providers do: ${providersWithSpeakers().join(", ")}.`, ); } // A request routed where the speaker option is ignored is answered without speakers. From df8361b91471b99de28bf87c7481ed7dd81d5f3e Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 11:56:02 +0100 Subject: [PATCH 2/8] test(transcription): Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter --- src/services/TranscriptionService.test.ts | 112 ++++++++++++++++++++ src/ui/settings/PodNotesSettingsTab.test.ts | 28 ++++- 2 files changed, 139 insertions(+), 1 deletion(-) diff --git a/src/services/TranscriptionService.test.ts b/src/services/TranscriptionService.test.ts index d64d82ac..96390628 100644 --- a/src/services/TranscriptionService.test.ts +++ b/src/services/TranscriptionService.test.ts @@ -1269,6 +1269,8 @@ describe("TranscriptionService", () => { "x-ai/grok-stt-1.0": "2026-11-15", "elevenlabs/scribe-v2": null, "microsoft/mai-transcribe-2": null, + "mistralai/voxtral-mini-transcribe-3-0": null, + "qwen/qwen3-asr-1.7b": null, }); /** Answers the catalog with `listed`, and each transcription request with `answer`. */ const openRouter = ( @@ -1485,6 +1487,116 @@ describe("TranscriptionService", () => { expect(note).toContain(`Part ${partBytes.length}.`); }); + describe("models without speaker labels", () => { + const VOXTRAL = "mistralai/voxtral-mini-transcribe-3-0"; + const QWEN = "qwen/qwen3-asr-1.7b"; + /** Voxtral's verbose_json: timed segments and no words. */ + const segmentsAnswer = { + text: "Hello there. Hi!", + duration: 11, + segments: [ + { id: 0, start: 1.2, end: 1.9, text: " Hello there." }, + { id: 1, start: 10.16, end: 10.5, text: " Hi!" }, + ], + }; + const wordsAnswer = { + ...answers, + words: answers.words.map(({ speaker: _s, speaker_label: _l, ...rest }) => rest), + }; + + test.each([ + [ + VOXTRAL, + "Voxtral Mini Transcribe 3", + segmentsAnswer, + { model: VOXTRAL, response_format: "verbose_json" }, + ], + [ + QWEN, + "Qwen3 ASR 1.7B", + wordsAnswer, + { + model: QWEN, + response_format: "verbose_json", + "timestamp_granularities[]": "word", + }, + ], + ])( + "%s gets the upload as its measured multipart request, and with speaker labels on the note is timed, unlabeled, and the notice says why", + async (model, label, answer, expected) => { + openRouter(() => Response.json(answer)); + const plugin = useOpenRouter({ + model, + feeds: savedFeedRepository(publicFeed, [mockEpisode]), + }); + + await expect(finalNotice(plugin)).resolves.toBe( + `Saved without speaker labels, because ${label} does not label speakers. These providers do: ElevenLabs, Deepgram, OpenRouter.`, + ); + + expect(sent()).toHaveLength(1); + const [request] = sent(); + expect(request.headers).toEqual({ Authorization: "Bearer openrouter-key" }); + expect(await fields(request)).toEqual(expected); + expect(new Uint8Array(await (await sentFile(request)).arrayBuffer())).toEqual( + audio, + ); + expect(savedNote(plugin)).toBe(unlabeled); + }, + ); + + test("Voxtral's segments from two parts of at most 900 s become paragraphs timed from the episode's start", async () => { + const stream = MP3_STREAMS.mpeg25At8Kbps; + const plugin = useOpenRouter({ model: VOXTRAL, speakers: false }); + getEpisodeAudioBufferMock.mockResolvedValue({ + buffer: mp3Bytes(stream, Math.ceil(1500 / stream.frameSeconds)).buffer, + extension: "mp3", + }); + const partBytes: number[] = []; + openRouter(async (request) => { + partBytes.push((await sentFile(request)).size); + const part = partBytes.length; + return Response.json({ + text: `Part ${part} opens. Part ${part} ends.`, + segments: [ + { id: 0, start: 2.5, end: 6, text: ` Part ${part} opens.` }, + { id: 1, start: 40, end: 43.25, text: ` Part ${part} ends.` }, + ], + }); + }); + + await expect(finalNotice(plugin)).resolves.toBe( + "Transcription completed and saved.", + ); + + expect(partBytes).toHaveLength(2); + for (const bytes of partBytes) { + expect(mp3Seconds(stream, bytes)).toBeLessThanOrEqual(900); + } + const second = mp3Seconds(stream, partBytes[0]); + const note = savedNote(plugin); + expect( + [...note.matchAll(/time=([\d.]+)\) (.+)$/gm)].map(([, t, s]) => [t, s]), + ).toEqual([ + ["2.5", "Part 1 opens."], + ["40", "Part 1 ends."], + [`${Math.round((second + 2.5) * 1000) / 1000}`, "Part 2 opens."], + [`${Math.round((second + 40) * 1000) / 1000}`, "Part 2 ends."], + ]); + }); + + test("an answer with neither words nor segments writes no note", async () => { + openRouter(() => Response.json({ text: "Hello there.", duration: 11 })); + const plugin = useOpenRouter({ model: VOXTRAL, speakers: false }); + + await expect(finalNotice(plugin)).resolves.toBe( + "Transcription failed: OpenRouter found no speech in this episode.", + ); + + expect(plugin.app.vault.create).not.toHaveBeenCalled(); + }); + }); + describe("speaker labels across the parts of an episode", () => { const stream = MP3_STREAMS.mpeg25At8KbpsCrcPadded; const VOICES = ["", "Ann", "Bob", "Cy"]; diff --git a/src/ui/settings/PodNotesSettingsTab.test.ts b/src/ui/settings/PodNotesSettingsTab.test.ts index 47e6b649..5de11402 100644 --- a/src/ui/settings/PodNotesSettingsTab.test.ts +++ b/src/ui/settings/PodNotesSettingsTab.test.ts @@ -459,7 +459,7 @@ describe("Transcripts", () => { : new Response("", { status: 404 }), ); - test("choosing it shows its API key row and its three models, Grok STT 2.0 recommended", async () => { + test("choosing it shows its API key row and its five models, Grok STT 2.0 recommended", async () => { const { provider, model, keyButtons, disk } = openTranscripts(); choose(provider(), "openrouter"); @@ -471,11 +471,37 @@ describe("Transcripts", () => { ["x-ai/grok-stt-2.0", "Grok STT 2.0"], ["elevenlabs/scribe-v2", "ElevenLabs Scribe v2"], ["microsoft/mai-transcribe-2", "MAI-Transcribe 2"], + ["mistralai/voxtral-mini-transcribe-3-0", "Voxtral Mini Transcribe 3"], + ["qwen/qwen3-asr-1.7b", "Qwen3 ASR 1.7B"], ]); expect(model()).toHaveDisplayValue("Recommended (Grok STT 2.0)"); expect((await disk()).transcript).toMatchObject({ provider: "openrouter", model: "" }); }); + test.each([ + ["mistralai/voxtral-mini-transcribe-3-0", "Voxtral Mini Transcribe 3"], + ["qwen/qwen3-asr-1.7b", "Qwen3 ASR 1.7B"], + ])( + "choosing %s says Speaker labels are not available, and choosing Grok STT 2.0 again lifts that", + async (id, label) => { + const { ui, model, speakerLabels } = openTranscripts( + transcript({ provider: "openrouter", ...withSpeakerLabels(true) }), + ); + const reason = `${label} does not label speakers.`; + await settled(); + + choose(model(), id); + + expect(ui.getByText(reason)).toBeVisible(); + expect(speakerLabels()).toBeChecked(); + expect(speakerLabels()).toBeEnabled(); + + choose(model(), "x-ai/grok-stt-2.0"); + + expect(ui.queryByText(reason)).toBeNull(); + }, + ); + test("the catalog read as the row draws leaves out a model it does not list and dates one that ends", async () => { listing({ "x-ai/grok-stt-2.0": "2099-01-31", "microsoft/mai-transcribe-2": null }); const { model } = openTranscripts(transcript({ provider: "openrouter" })); From 7e62c77e0f9269d5613516be60e6e761014b6c12 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 11:56:02 +0100 Subject: [PATCH 3/8] feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter Voxtral answers timed segments and no words, so a segments answer becomes cues timed from the episode's start, as words do. Both models take parts of 12 MB and 900 s, because 20 MB took 33 to 35 s to answer. --- src/services/transcription/cues.ts | 35 +++++++----- .../transcription/providers/openrouter.ts | 55 ++++++++++++++++--- src/services/transcription/providers/types.ts | 15 ++++- 3 files changed, 82 insertions(+), 23 deletions(-) diff --git a/src/services/transcription/cues.ts b/src/services/transcription/cues.ts index e83558f1..27124e89 100644 --- a/src/services/transcription/cues.ts +++ b/src/services/transcription/cues.ts @@ -4,7 +4,7 @@ import { type TranscriptCue, untimed, } from "src/services/publishedTranscript/parse"; -import type { PartTranscript, Word } from "./providers/types"; +import type { PartTranscript, Segment, Word } from "./providers/types"; /** What one part of the episode gave back. `transcript` is absent when its request failed. */ export interface TranscribedPart { @@ -13,7 +13,8 @@ export interface TranscribedPart { readonly transcript?: PartTranscript; } -type TimedPart = TranscribedPart & { readonly transcript?: { readonly words: readonly Word[] } }; +type TimedTranscript = Exclude; +type TimedPart = TranscribedPart & { readonly transcript?: TimedTranscript }; const PARAGRAPH_LENGTH = 500; @@ -24,10 +25,10 @@ function failedPartMarker(index: number): string { /** * The episode's transcript from its parts in order, or undefined when no part holds speech. - * Words become one cue each, timed from the start of the episode, and the renderer folds - * them into paragraphs as it does for a published word-level transcript. Text from a model - * that returns no times becomes untimed paragraphs. Without `speakers` no cue names one, - * whatever the provider answered. + * Words, or segments from a model that times no single word, become one cue each, timed from + * the start of the episode, and the renderer folds them into paragraphs as it does for a + * published transcript. Text from a model that returns no times becomes untimed paragraphs. + * Without `speakers` no cue names one, whatever the provider answered. */ export function transcriptFromParts( parts: readonly TranscribedPart[], @@ -45,11 +46,17 @@ export function transcriptFromParts( function textOf(transcript: PartTranscript): string { return "text" in transcript ? transcript.text - : transcript.words.map((word) => word.text).join(" "); + : timesOf(transcript) + .map((item) => item.text) + .join(" "); +} + +function timesOf(transcript: TimedTranscript): readonly (Word | Segment)[] { + return "words" in transcript ? transcript.words : transcript.segments; } function isTimed(parts: readonly TranscribedPart[]): parts is readonly TimedPart[] { - return parts.every(({ transcript }) => !transcript || "words" in transcript); + return parts.every(({ transcript }) => !transcript || !("text" in transcript)); } /** @@ -84,14 +91,14 @@ function cuesOf( }; return parts.flatMap(({ start, transcript }, index): TranscriptCue[] => transcript - ? transcript.words.map((word) => ({ - start: toMilliseconds(start + word.start), - end: toMilliseconds(start + word.end), + ? timesOf(transcript).map((item) => ({ + start: toMilliseconds(start + item.start), + end: toMilliseconds(start + item.end), speaker: - speakers && word.speaker !== undefined - ? numberOf(index, word.speaker) + speakers && "speaker" in item && item.speaker !== undefined + ? numberOf(index, item.speaker) : undefined, - text: word.text, + text: item.text, })) : [{ start: toMilliseconds(start), text: failedPartMarker(index) }], ); diff --git a/src/services/transcription/providers/openrouter.ts b/src/services/transcription/providers/openrouter.ts index 594c4946..ddab3dea 100644 --- a/src/services/transcription/providers/openrouter.ts +++ b/src/services/transcription/providers/openrouter.ts @@ -9,6 +9,7 @@ import type { PartTranscript, Provider, ProviderRequest, + Segment, Word, } from "./types"; @@ -20,6 +21,8 @@ const CATALOG_URL = "https://openrouter.ai/api/v1/models?output_modalities=trans * 1,980 s were answered in 20 to 31 s, so these leave room. */ const PART_CAPS = { maxUploadBytes: 18_000_000, maxSeconds: 1800 }; +/** Voxtral and Qwen3 ASR took 33 to 35 s to answer 20 MB, so they get smaller parts. */ +const SLOW_PART_CAPS = { maxUploadBytes: 12_000_000, maxSeconds: 900 }; type Body = (input: PartInput) => Promise; @@ -51,10 +54,10 @@ const grok: Body = async ({ model, speakers, audio }) => { }; /** - * Scribe v2 and MAI-Transcribe 2 take `diarize` beside the model, and list words only when - * asked for speakers or word times. Only Scribe v2 fetches an address. + * Scribe v2, MAI-Transcribe 2 and Qwen3 ASR list words only when asked for word times or + * speakers. The first two take `diarize` beside the model. Only Scribe v2 fetches an address. */ -const topLevelDiarize: Body = async ({ model, speakers, audio }) => { +const wordTimes: Body = async ({ model, speakers, audio }) => { if ("url" in audio) { return { model: model.id, @@ -73,6 +76,16 @@ const topLevelDiarize: Body = async ({ model, speakers, audio }) => { return form; }; +/** Voxtral refuses a request for word times with a 400, and times its segments unasked. */ +const segmentTimes: Body = async ({ model, audio }) => { + if ("url" in audio) throw new Error("Voxtral takes no audio address."); + const form = new FormData(); + form.append("model", model.id); + form.append("file", audio.file); + form.append("response_format", "verbose_json"); + return form; +}; + const MODELS: NonEmpty = [ { id: "x-ai/grok-stt-2.0", @@ -91,7 +104,7 @@ const MODELS: NonEmpty = [ ...PART_CAPS, // Its reel heard 4 voices on a show with 10. mergeSpeakers: false, - body: topLevelDiarize, + body: wordTimes, }, { id: "microsoft/mai-transcribe-2", @@ -100,7 +113,23 @@ const MODELS: NonEmpty = [ url: false, ...PART_CAPS, mergeSpeakers: false, - body: topLevelDiarize, + body: wordTimes, + }, + { + id: "mistralai/voxtral-mini-transcribe-3-0", + label: "Voxtral Mini Transcribe 3", + speakers: false, + url: false, + ...SLOW_PART_CAPS, + body: segmentTimes, + }, + { + id: "qwen/qwen3-asr-1.7b", + label: "Qwen3 ASR 1.7B", + speakers: false, + url: false, + ...SLOW_PART_CAPS, + body: wordTimes, }, ]; @@ -157,9 +186,13 @@ export const openrouter: Provider = { }, }; +/** Words when the answer lists them, else its segments. An answer with neither holds no speech. */ function parse(answer: unknown): PartTranscript { - const words = isPlainObject(answer) ? answer.words : undefined; - if (!Array.isArray(words)) return { words: [] }; + if (!isPlainObject(answer)) return { words: [] }; + const { words, segments } = answer; + if (!Array.isArray(words)) { + return { segments: Array.isArray(segments) ? segments.flatMap(segmentOf) : [] }; + } return { words: words.flatMap((item: unknown): Word[] => { // Scribe v2 also lists sounds, such as "[gentle music]", as `audio_event` items. @@ -182,3 +215,11 @@ function parse(answer: unknown): PartTranscript { }), }; } + +function segmentOf(item: unknown): Segment[] { + if (!isPlainObject(item)) return []; + const { text, start, end } = item; + return typeof text === "string" && typeof start === "number" && typeof end === "number" + ? [{ text, start, end }] + : []; +} diff --git a/src/services/transcription/providers/types.ts b/src/services/transcription/providers/types.ts index bdbcb62b..53cc696d 100644 --- a/src/services/transcription/providers/types.ts +++ b/src/services/transcription/providers/types.ts @@ -33,8 +33,19 @@ export interface Word { readonly speaker?: string; } -/** What one request gave back: timed words, or text from a model that returns no times. */ -export type PartTranscript = { readonly words: readonly Word[] } | { readonly text: string }; +/** A stretch of speech, such as a sentence, from a model that times no single word. */ +export interface Segment { + readonly text: string; + /** Seconds from the start of the audio this request carried. */ + readonly start: number; + readonly end: number; +} + +/** What one request gave back: timed words or segments, or text from a model that returns no times. */ +export type PartTranscript = + | { readonly words: readonly Word[] } + | { readonly segments: readonly Segment[] } + | { readonly text: string }; export type AudioInput = { readonly url: PublicAudioUrl } | { readonly file: File }; From 0b032d8ed0fd59716c36140606bc7bee50cc9cd6 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 11:59:28 +0100 Subject: [PATCH 4/8] test(e2e): Voxtral Mini Transcribe 3 and Qwen3 ASR upload as multipart and save timed notes The OpenRouter stub answers them as they were measured: Voxtral with segments, or a 400 when asked for word times, and Qwen3 ASR with words and no speakers. --- tests/e2e/journey-resilience.test.ts | 51 ++++++++++++++++++++++++++-- tests/e2e/journey/fixtures.ts | 34 +++++++++++++++---- 2 files changed, 76 insertions(+), 9 deletions(-) diff --git a/tests/e2e/journey-resilience.test.ts b/tests/e2e/journey-resilience.test.ts index 87e7ed68..cf1b20ac 100644 --- a/tests/e2e/journey-resilience.test.ts +++ b/tests/e2e/journey-resilience.test.ts @@ -894,11 +894,15 @@ describe("resilience journeys", () => { describe("OpenRouter", () => { const secretId = "podnotes-openrouter-api-key-464646"; - /** Seeds `show` as playing `spoken` on OpenRouter's `model`, with a key and the stub. */ + /** + * Seeds `show` as playing `spoken` on OpenRouter's `model`, with a key, speaker labels + * on and the stub. The note names the speakers only when `labels`. + */ const transcribeWithOpenRouter = async ( show: FixtureFeed, spoken: FixtureEpisode, model: string, + labels = true, ) => { const context = getContext(); const { obsidian, sandbox } = context; @@ -941,8 +945,8 @@ describe("resilience journeys", () => { ), ).toBe( `# Spoken Episode\n\nPodcast: ${show.title}\nDate: \n\n` + - `[00:00:00](${link(0)}) **1:** Journey transcript opens here.\n\n` + - `[00:00:12](${link(12)}) **2:** A second speaker answers.`, + `[00:00:00](${link(0)}) ${labels ? "**1:** " : ""}Journey transcript opens here.\n\n` + + `[00:00:12](${link(12)}) ${labels ? "**2:** " : ""}A second speaker answers.`, ); expect(await openRouterCatalogReads(obsidian)).toEqual([null]); return server; @@ -971,6 +975,7 @@ describe("resilience journeys", () => { contentType: "application/json", model: "elevenlabs/scribe-v2", diarize: true, + granularities: ["word"], url: server.mediaUrl(show, spoken), audioBytes: 0, sha256: null, @@ -1003,12 +1008,52 @@ describe("resilience journeys", () => { contentType: "application/json", model: "x-ai/grok-stt-2.0", diarize: true, + granularities: ["word", "segment"], url: null, audioBytes: audio.byteLength, sha256: createHash("sha256").update(audio).digest("hex"), }, ]); }, 60_000); + + test.each([ + ["J-TR-11", "mistralai/voxtral-mini-transcribe-3-0", "voxtral", []], + ["J-TR-12", "qwen/qwen3-asr-1.7b", "qwen", ["word"]], + ])( + "%s %s gets the upload as multipart, and its note is timed without speaker labels", + async (_id, model, slug, granularities) => { + const { obsidian } = getContext(); + const spoken: FixtureEpisode = { + title: "Spoken Episode", + seconds: 4, + pubDate: pubDate(0), + }; + const show: FixtureFeed = { + title: `Journey OpenRouter ${slug}`, + slug: `openrouter-${slug}`, + artwork: true, + episodes: [spoken], + }; + + const server = await transcribeWithOpenRouter(show, spoken, model, false); + + const audio = server.audio(show, spoken); + expect(await openRouterRequests(obsidian)).toEqual([ + { + method: "POST", + authorization: "Bearer sk-or-journey-test", + contentType: expect.stringMatching(/^multipart\/form-data; boundary=/), + model, + diarize: false, + granularities, + url: null, + audioBytes: audio.byteLength, + sha256: createHash("sha256").update(audio).digest("hex"), + }, + ]); + }, + 60_000, + ); }); test("J-TR-05 feed text cannot give the transcript note a block ID or an inline field", async () => { diff --git a/tests/e2e/journey/fixtures.ts b/tests/e2e/journey/fixtures.ts index e4da9deb..4d6fd070 100644 --- a/tests/e2e/journey/fixtures.ts +++ b/tests/e2e/journey/fixtures.ts @@ -606,6 +606,8 @@ export type OpenRouterRequest = Readonly<{ contentType: string | null; model: string | null; diarize: boolean; + /** The `timestamp_granularities` the request asked for. */ + granularities: string[]; /** The address the request named in place of the audio. */ url: string | null; audioBytes: number; @@ -619,7 +621,8 @@ const OPENROUTER_CATALOG_URL = /** * Answers OpenRouter's transcriptions with `first` from one speaker at 0 s and `second` from - * another 12 s in, and its catalog with the three models PodNotes offers. + * another 12 s in, and its catalog with the five models PodNotes offers. Qwen3 ASR answers + * words without speakers, and Voxtral segments, or a 400 when asked for word times, as they do. */ export async function installOpenRouterStub( obsidian: ObsidianClient, @@ -636,22 +639,38 @@ export async function installOpenRouterStub( const start = from + index * 0.4; return { word, start, end: start + 0.3, speaker, speaker_label: "speaker_" + speaker, confidence: 0.99 }; }); - const words = [...spoken(${jsLiteral(first)}, 0, 0), ...spoken(${jsLiteral(second)}, 1, 12)]; - const reply = (replyChannel, answer) => { + const opening = spoken(${jsLiteral(first)}, 0, 0); + const answering = spoken(${jsLiteral(second)}, 1, 12); + const words = [...opening, ...answering]; + const text = ${jsLiteral(`${first} ${second}`)}; + const segments = [ + { id: 0, start: 0, end: opening.at(-1).end, text: " " + ${jsLiteral(first)} }, + { id: 1, start: 12, end: answering.at(-1).end, text: " " + ${jsLiteral(second)} }, + ]; + const answers = { + "mistralai/voxtral-mini-transcribe-3-0": (granularities) => + granularities.length > 0 + ? [400, { error: { message: "Provider returned 400", code: 400 } }] + : [200, { text, segments }], + "qwen/qwen3-asr-1.7b": () => [200, { text, words: words.map(({ word, start, end }) => ({ word, start, end })) }], + }; + const reply = (replyChannel, answer, status = 200) => { const body = new TextEncoder().encode(JSON.stringify(answer)).buffer; - ipc.emit(replyChannel, {}, { status: 200, headers: { "content-type": "application/json" }, body }); + ipc.emit(replyChannel, {}, { status, headers: { "content-type": "application/json" }, body }); }; const answer = async (replyChannel, request) => { const headers = new Headers(request.headers); const contentType = request.contentType ?? null; let model = null; let diarize = false; + let granularities = []; let url = null; let bytes = null; if (contentType === "application/json") { const body = JSON.parse(new TextDecoder().decode(request.body)); model = body.model; diarize = body.diarize === true || body.provider?.options?.xai?.diarize === true; + granularities = body.timestamp_granularities ?? []; url = body.input_audio?.url ?? null; if (body.input_audio?.data) bytes = Uint8Array.from(atob(body.input_audio.data), (char) => char.charCodeAt(0)); } else { @@ -659,6 +678,7 @@ export async function installOpenRouterStub( const file = form.get("file"); model = form.get("model"); diarize = form.get("diarize") === "true"; + granularities = form.getAll("timestamp_granularities[]"); if (file instanceof Blob) bytes = new Uint8Array(await file.arrayBuffer()); } requests.push({ @@ -667,11 +687,13 @@ export async function installOpenRouterStub( contentType, model, diarize, + granularities, url, audioBytes: bytes?.byteLength ?? 0, sha256: bytes && require("node:crypto").createHash("sha256").update(bytes).digest("hex"), }); - reply(replyChannel, { text: ${jsLiteral(`${first} ${second}`)}, words }); + const [status, answer] = answers[model]?.(granularities) ?? [200, { text, words }]; + reply(replyChannel, answer, status); }; ipc.send = function (channel, replyChannel, request, ...rest) { if (channel === "request-url" && request?.url === ${jsLiteral(OPENROUTER_TRANSCRIPTIONS_URL)}) { @@ -680,7 +702,7 @@ export async function installOpenRouterStub( } if (channel === "request-url" && request?.url === ${jsLiteral(OPENROUTER_CATALOG_URL)}) { catalogReads.push(new Headers(request.headers).get("authorization")); - queueMicrotask(() => reply(replyChannel, { data: ["x-ai/grok-stt-2.0", "elevenlabs/scribe-v2", "microsoft/mai-transcribe-2"].map((id) => ({ id, expiration_date: null })) })); + queueMicrotask(() => reply(replyChannel, { data: ["x-ai/grok-stt-2.0", "elevenlabs/scribe-v2", "microsoft/mai-transcribe-2", "mistralai/voxtral-mini-transcribe-3-0", "qwen/qwen3-asr-1.7b"].map((id) => ({ id, expiration_date: null })) })); return; } return originalSend.call(ipc, channel, replyChannel, request, ...rest); From 41089b7464ad325fa7db88d00c123463209ae0e0 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 12:00:10 +0100 Subject: [PATCH 5/8] build(size): raise the gzip budget by the 211 bytes Voxtral and Qwen3 ASR measure The branch this builds on leaves main.js at 106,873 gzip bytes, built with the same vite command from an export of its head. The two OpenRouter models and reading Voxtral's segment times add 211, which takes it to 107,084 and over the 106,970 budget. The budget rises by those 211 bytes to 107,181, so the 97 bytes of headroom stay as they were. --- scripts/size-baseline.md | 12 +++++++++++- scripts/size.mjs | 2 +- scripts/size.test.ts | 2 +- 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/scripts/size-baseline.md b/scripts/size-baseline.md index 005dce57..6fcad6c9 100644 --- a/scripts/size-baseline.md +++ b/scripts/size-baseline.md @@ -12,7 +12,7 @@ main.js raw: 307131 bytes; budget 330000 bytes; 22869 bytes remaining main.js gzip (level 9): 97161 bytes; budget 100000 bytes; 2839 bytes remaining ``` -The gzip budget is now 106,970 bytes. On 2026-10-14 it rose from 100,000 for +The gzip budget is now 107,181 bytes. On 2026-10-14 it rose from 100,000 for the full player's Transcript panel, which the owner approved at about 1.9 KB and which measured 2,576 bytes. The coordinator set 103,000 to keep the panel's Find and Follow chip and to leave room for fixes in flight. The @@ -64,6 +64,16 @@ main.js raw: 327446 bytes; budget 330000 bytes; 2554 bytes remaining main.js gzip (level 9): 106873 bytes; budget 106970 bytes; 97 bytes remaining ``` +Later on 2026-11-14 it rose from 106,970 for Voxtral Mini Transcribe 3 and +Qwen3 ASR 1.7B on OpenRouter. The branch it builds on measured 106,873 bytes. +The two models and the segment times Voxtral answers with measured 211 more. +The budget rose by those 211 bytes. With them: + +```text +main.js raw: 328174 bytes; budget 330000 bytes; 1826 bytes remaining +main.js gzip (level 9): 107084 bytes; budget 107181 bytes; 97 bytes remaining +``` + Gzip uses Node's `gzipSync` at level 9, without a filename or timestamp. Budgets use decimal bytes. The brief's 96,533-byte reference is unexplained. This report fixes the compressor instead of comparing compressed sizes across diff --git a/scripts/size.mjs b/scripts/size.mjs index ecd93c8f..21f885e4 100644 --- a/scripts/size.mjs +++ b/scripts/size.mjs @@ -10,7 +10,7 @@ try { if (!bundle.length) throw new Error(`${path} is empty. Run npm run build first.`); const measurements = [ { name: "raw", bytes: bundle.length, budget: 330_000 }, - { name: "gzip (level 9)", bytes: gzipSync(bundle, { level: 9 }).length, budget: 106_970 }, + { name: "gzip (level 9)", bytes: gzipSync(bundle, { level: 9 }).length, budget: 107_181 }, ]; const rows = measurements.map(({ name, bytes, budget }) => { const overage = Math.max(0, bytes - budget); diff --git a/scripts/size.test.ts b/scripts/size.test.ts index b3f1e93f..0e4d848e 100644 --- a/scripts/size.test.ts +++ b/scripts/size.test.ts @@ -30,7 +30,7 @@ describe("bundle budget CLI", () => { const result = check("gzip.js", randomBytes(110_000)); expect(result.status).toBe(1); expect(result.stdout).toMatch( - /gzip \(level 9\): \d+ bytes; budget 106970 bytes; OVER by \d+ bytes/, + /gzip \(level 9\): \d+ bytes; budget 107181 bytes; OVER by \d+ bytes/, ); expect(result.stdout).toContain("budget 330000 bytes; 220000 bytes remaining"); }); From 979f2bb544e67f500e7e976bafac38490c42fb28 Mon Sep 17 00:00:00 2001 From: Christian Bager Bach Houmann Date: Sat, 14 Nov 2026 12:06:27 +0100 Subject: [PATCH 6/8] test(verify): answer Voxtral Mini Transcribe 3 and Qwen3 ASR as they were measured The fixture provider answers Voxtral with segments, or a 400 when asked for word times, and Qwen3 ASR with words and no speakers. The transcripts recipe covers both on a 70 min MP3. --- .claude/skills/verify/SKILL.md | 3 ++- .claude/skills/verify/features/transcripts.md | 5 +++-- .claude/skills/verify/net.mjs | 21 ++++++++++++++++--- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/.claude/skills/verify/SKILL.md b/.claude/skills/verify/SKILL.md index 363eb1e6..2070e31d 100644 --- a/.claude/skills/verify/SKILL.md +++ b/.claude/skills/verify/SKILL.md @@ -229,7 +229,8 @@ its profile. It keeps the vault and `.obsidian-e2e-artifacts/`. Delete provider while it watches. It decodes Grok's MP3 uploads with Chromium's decoder and hears a tone under 500 Hz as one speaker and a higher tone as another, and logs each upload's `decodedSeconds` and `heardSpeakers`. Audio - it cannot decode is answered with a 400. + it cannot decode is answered with a 400. It answers Voxtral with segments, or + a 400 when asked for word times, and Qwen3 ASR with words and no speakers. - `.claude/skills/verify/ui.mjs` sends trusted input through `obsidian dev:cdp`. Its verbs are `click [--text ] [--right]`, diff --git a/.claude/skills/verify/features/transcripts.md b/.claude/skills/verify/features/transcripts.md index a6e1fddc..bdd46de4 100644 --- a/.claude/skills/verify/features/transcripts.md +++ b/.claude/skills/verify/features/transcripts.md @@ -18,7 +18,7 @@ from `Transcript template`, and opens. - `transcript-settings` shows `Transcription provider`, the chosen provider's API key row alone, `Model`, `Speaker labels` and, while those are on, `Speaker label format`. `Speaker labels` is disabled for a model without them. - `transcript-elevenlabs` writes timestamped paragraphs with numbered speakers from one `scribe_v3` request. ElevenLabs gets the address of an episode a saved public feed lists, and the audio itself for a private feed's episode, a local file, and an address that carries a credential or is not public. - `transcript-deepgram` does the same with `nova-3`. -- `transcript-openrouter` offers `Grok STT 2.0` (recommended), `ElevenLabs Scribe v2` and `MAI-Transcribe 2`, filtered by OpenRouter's public model catalog. Grok gets uploads as base64 JSON routed to xAI, Scribe v2 the address of a public episode, and an upload goes in parts of at most 18 MB and 1,800 s. +- `transcript-openrouter` offers `Grok STT 2.0` (recommended), `ElevenLabs Scribe v2`, `MAI-Transcribe 2`, `Voxtral Mini Transcribe 3` and `Qwen3 ASR 1.7B`, filtered by OpenRouter's public model catalog. Grok gets uploads as base64 JSON routed to xAI, Scribe v2 the address of a public episode, and an upload goes in parts of at most 18 MB and 1,800 s, or 12 MB and 900 s for Voxtral and Qwen3 ASR, which label no speakers. - `transcript-openai` sends OpenAI the audio as `gpt-transcribe` uploads of at most 24 MB and 2,070 s, one at a time, and writes paragraphs without timestamps or speaker labels. - `transcript-existing` refuses a second run while the note exists. @@ -39,7 +39,7 @@ Preconditions: - **Published.** Play `VTT Episode` ([player.md](./player.md)), then run `UI key Ctrl+p`, `UI type .prompt-input 'Transcribe current episode'` and `UI key Enter`. The exact command is the first suggestion; `--text 'Transcribe current episode'` matches both transcribe commands. The notices read `Queued "VTT Episode" for transcription. It will start automatically.` and, in the progress notice headed `Transcription: VTT Episode (00:00:00):`, `Saved the transcript the feed publishes.` `transcripts/PodNotes Transcripts/VTT Episode.md` opens. Its body starts `[00:00:00](obsidian://podnotes?episodeName=VTT%20Episode&url=...&time=0.48) From Potsdam in Brandenburg, Germany,`. `$NET log` lists `GET .../t/podnews.vtt` and nothing at `api.elevenlabs.io`, `api.deepgram.com` or `api.openai.com`. `node .claude/skills/verify/fixtures.mjs requests` has `/t/podnews.vtt`, not `/t/buzzcast.html`. Repeat with `SRT Episode` and `JSON Episode`. - **Seek.** Switch to Reading view and click a timestamp. `api.currentTime` lands on its time. -- **Settings.** Open Settings > PodNotes. Under `Transcripts` a new vault shows `Transcription provider` on `ElevenLabs`, `ElevenLabs API key`, `Model` on `Recommended (Scribe v3)`, `Speaker labels` on, `Speaker label format`, then the path and template. Choosing `Deepgram`, `OpenRouter` or `OpenAI` swaps the key row for that provider's and the model list for its models. On `OpenRouter` the `Model` row offers `Recommended (Grok STT 2.0)`, `Grok STT 2.0`, `ElevenLabs Scribe v2` and `MAI-Transcribe 2`, and `$NET log` shows one `GET https://openrouter.ai/api/v1/models` with no `Authorization`. A model the catalog dates reads `