[Ved-Padmawar]

Summary

Closes #240

This PR adds word-level timestamps to Whisper through TRANSCRIBE_TIMESTAMPS_WORD.

Whisper currently provides segment-level timestamps, which aren't precise enough for streaming use cases like LocalAgreement. When trimming committed audio, we need to know exactly where a word ends rather than relying on the timestamp of an entire segment.

To address this, I've added a DTW-based alignment approach similar to what's used in OpenAI's timing.py and whisper.cpp.

After each 30-second window is decoded, an additional forced decoder pass extracts cross-attention from the alignment heads. The attention weights are then median-filtered and aligned with audio frames using DTW to estimate individual word boundaries.

The feature is opt-in and uses the existing timestamp enum and word accessors, so no public API or ABI changes are needed.

For now, word-level timestamps are supported on standard OpenAI Whisper models with known alignment-head presets. Other checkpoints will continue using segment-level timestamps.

Scope

• Added alignment.{h,cpp} under src/arch/whisper for DTW-based word alignment.
• Updated the Whisper decoder and model to support word-level timestamps.
• Added unit tests and real-model smoke tests.
• Updated the documentation.
• Added the appropriate MIT license attribution for the alignment-head presets sourced from whisper.cpp.

AI Assistance

I used AI to help with parts of the implementation, tests, and documentation. I've reviewed the changes, tested the implementation, and take responsibility for the final code.

Validation

Tested word-level timestamps on the following models using CPU:

• tiny
• base
• small
• medium
• large-v3-turbo
• large-v3

All new unit tests and smoke tests pass.

On whisper-base, Vulkan output matches CPU. CUDA and Metal haven't been tested yet.

The existing text decoding path remains unchanged, so this shouldn't affect transcription accuracy or WER.

Example output

Using whisper-base with samples/jfk.wav and --timestamps word:

words: 22
  [   0.00 ->    0.52] And
  [   0.52 ->    0.82] so
  [   0.82 ->    1.18] my
  [   1.18 ->    1.56] fellow
  [   1.56 ->    2.10] Americans,
  [   3.18 ->    3.74] ask
  [   3.74 ->    4.24] not
  [   4.24 ->    5.52] what
  [   5.52 ->    5.76] your
  [   5.76 ->    6.26] country
  [   6.26 ->    6.62] can
  [   6.62 ->    6.82] do
  [   6.82 ->    7.06] for
  [   7.06 ->    7.34] you,
  [   7.74 ->    8.48] ask
  [   8.48 ->    8.76] what
  [   8.76 ->    9.02] you
  [   9.02 ->    9.32] can
  [   9.32 ->    9.56] do
  [   9.56 ->    9.78] for
  [   9.78 ->    9.94] your
  [   9.94 ->   10.36] country.