[bgotthold-aai]

Port of the Python SDK's Text-to-Speech clients (assemblyai-python-sdk #277 and #279) to this SDK, following its conventions: a service on the client, snake_case request bodies, camelCase session parameters, the streaming client's on() listener pattern and WebSocket polyfill, jest-fetch-mock and jest-websocket-mock tests, a CHANGELOG entry and a minor version bump (4.42.0).

What

client.tts, a TextToSpeechService on the new ttsBaseUrl (https://tts.us.assemblyai.com by default; https://tts.eu.assemblyai.com for the EU zone):

• synthesize(request) posts POST /v1/speech buffered and returns a SpeechResponse: the complete audio (Uint8Array) plus the response headers as requestId, region, model, voiceId, language, encoding, sampleRate, container, bitRate, contentType, voiceExpiresAt, billedCharacters, audioDurationMs. voiceId reads X-TTS-Voice-Id and, until the API sends it, is derived from X-TTS-Voice.
• stream(request) returns a TextToSpeechStream: an async iterable of audio chunks over the response's web ReadableStream (.body), with .info for the headers and .bytes() to collect it. The timeout covers the headers and first audio only; leaving the loop early cancels the body; a body cut after the first audio throws TextToSpeechError with retryable: true.
• session(params) returns a TextToSpeechSession for the WebSocket API (/v1/ws): connect() waits for Begin; speak(text) returns one request's audio as an async iterable; generate() / flush() / audio(flushId?) stream text in and pipeline requests (audio() with no id follows the request being written); cancel() ends the current iteration at once; keepAlive(); close() sends Terminate and resolves with the Termination summary. on("begin" | "audio" | "flushDone" | "wordBoundaries" | "cancelled" | "termination" | "error" | "close", fn) for push-style use; an audio listener switches audio from pull to push. A session the API ended can connect() again on the same object.
• createTemporaryToken({ expires_in_seconds }) mints a token on token.assemblyai.com for browser sessions, where the API key must not go (the same token-versus-header branching and browser warning as StreamingTranscriber).
• TextToSpeechError carries the API's envelope: status, errorCode, param, requestId (the session id on a session), retryable, retryAfter, errors, closeCode, body. A failure nothing observed in push mode rejects close().
• Types in src/types/tts, exported from the package root; README ## Text-To-Speech, CLAUDE.md Key APIs and section, samples/tts.ts.

Out of scope, as in Python: the OpenAI-compatible route, /v1/speech/events, the Voices API (voices are managed in the dashboard).

Tests

• tests/unit/tts.test.ts (12) and tests/unit/tts-session.test.ts (19) against jest-fetch-mock and a fake API over jest-websocket-mock that answers Generate/Flush/Cancel/Terminate the way the server does: bodies and headers, both error shapes, retry hints, the EU host, the token path, speaking, pipelining, barge-in before and after a request, a reader following the text being written, long-text splitting on code points, push mode, lost connections, reconnecting the same object, closing during a pending connect, body cancellation on an early break, and the headers-only stream timeout.
• Full unit suite passes apart from the three pre-existing utils.test.ts user-agent failures (they compare against the local Node version). pnpm run lint (eslint, tsc, prettier, publint) and pnpm run build are clean.
• Live against production from dist/node.mjs: a WAV with a RIFF header and matching duration, streamed 8 kHz mu-law, a warm session at 83 to 117 ms to first audio per sentence, barge-in with zero bytes after cancel(), word boundaries, a voice_not_found refusal as a typed error on both routes.
• A high-effort self-review before the push found ten issues (an off-by-one that left a reader following the text being written hanging after cancel(), the stream timeout covering body reads, no body cancellation on an early break, concurrent readers losing audio, a pending connect() outliving close(), UTF-16 splitting of long text, push-mode marker growth, a late pull guard in speak(), a close listener that never fired on client close, and an O(n) queue scan); all fixed with tests. The off-by-one also exists in the Python session and is fixed there separately.

🤖 Generated with Claude Code

[aleks-mitov]

Thanks Ben, this is a big and careful port, and a lot of it is already right: request bodies and frames match the API, the error envelope maps cleanly onto TextToSpeechError, flush and cancel share one id counter (and the cancel watermark is correct for the text being written), the pull queue drops only audio on overflow, the base URL path prefix is kept, and a mid-stream cut surfaces as a retryable SDK error. It builds and lints clean and works end to end against sandbox on every runtime I tried. It is not quite ready to merge: the majors are about stopping audio and about reconnects, which are exactly the voice-agent paths this is for. Most of the HTTP ones live in TextToSpeechStream and the deadline helper, so one pass there (lock the reader when the headers arrive, keep the caller's signal attached until the body ends, cancel through the active reader, add an idle read timeout, detach listeners) covers the GC, abort, cancel, read-timeout and listener-leak comments together.

Counts: 0 blockers, 6 majors, 27 minors, 11 nits (44 inline comments).

Majors

• stream() can return zero bytes with no error on Node 20 and 22 when the first read happens after an await: undici cancels the unlocked body once the Response is garbage collected. (src/services/tts/service.ts:295)
• stream() stops forwarding options.signal once the headers arrive, so aborting during playback does nothing. (src/services/tts/service.ts:32)
• TextToSpeechStream.cancel() is a silent no-op while a loop is reading (the body is locked). (src/services/tts/service.ts:200)
• audio() binds its flush id at the first next(), so an iterator created before flush() reads the wrong request and drops the intended one. (src/services/tts/session.ts:563)
• Reconnecting from the close listener makes a pending audio() end normally, hiding the cut. (src/services/tts/session.ts:593)
• A token session cannot reconnect (the token is spent or expired), but the docs and a test promise it can. (src/services/tts/session.ts:244)

Python review fixes, carried over or not

• Carried: audio dropped before markers on overflow (but see the enqueue() comment for the listener case), one counter for flush and cancel ids, close code from the received frame or Error frame, Begin limits read for generate(), base URL path prefix kept, container raw by default, voice_id only (the retired voice is never sent), X-TTS-Voice-Id read, a mid-stream cut raised as the SDK error, the 120 s inactivity close in the README, reconnect after the API ends a session (with an API key).
• Not yet: listener isolation, the send vs close race, token reconnects, the biometric consent errors, and on Node the long read timeout for synthesize(). See the inline comments.

What I verified

• pnpm build, lint:tsc, lint:eslint, lint:format and lint:publint all pass, with the same output and warnings as main. Unit tests on Node 18 and 20 (the CI matrix): 385/385, including the 31 new TTS tests, which also passed 10 of 10 repeated runs with no open handles. On Node 22, pnpm test:unit has 3 failures in tests/unit/utils.test.ts that fail on main too (Node 21+ defines navigator.userAgent), so not from this PR. typedoc adds one warning (TextToSpeechErrorFields, inline). Coverage of the new code: session.ts 88% of lines, service.ts 90%.
• Every build target exports the TTS classes and client.tts; browser.mjs and the UMD builds import no ws; a strict consumer type-checks against the declarations; samples/tts.ts type-checks. Bundle growth: node.mjs from 150.9 KB to 184.7 KB, the minified UMD from 15.6 KB to 19.5 KB gzipped.
• Live against sandbox with the built SDK (Node 18, 20 and 22, Bun, Deno, and the browser bundle on a WHATWG WebSocket): synthesize() for wav, raw A-law at 8 kHz, mp3 and 44.1 kHz with every header field matching the raw response; stream() bytes equal to a raw fetch; HTTP refusals (voice_not_found, single and multi-field invalid_parameter, the retired voice param, 401) mapped exactly; on sessions, Begin limits, speak(), pipelined flushes, cancel mid-audio, flush ids after cancels matching the server's, keepAlive(), push mode, the close() summary, an idle 3006 close followed by a reconnect with an API key, refusals before Begin (invalid_parameter, 1008, not_owner) with the close code from the frame, and token minting plus a token session. Against local fake servers: a body cut mid-stream gives the retryable TextToSpeechError, and the pull-queue overflow keeps the FlushDone markers.

Out of scope

• /v1/speech/events (word timestamps over HTTP) is not wrapped, same as the Python SDK, and X-Client-Request-Id/Server-Timing are not surfaced; both look like design calls.
• Not tested: a real browser engine, Cloudflare workerd, and React Native on a device (the React Native comment comes from an emulation).
• Patterns shared with StreamingTranscriber (unguarded listeners, no handshake User-Agent) are raised only where TTS adds something new.
• A few of these apply to the Python SDK too and are better fixed there separately: the flat terminate wait, the lazy audio() binding, stale audio after a cancel, speak() past the backlog, and re-reading a finished id. Python's cancel watermark (< and >) also leaves an audio() without an id hanging when cancel() takes its id, which this port gets right.

[aleks-mitov]

Major: stream() can return zero bytes with no error on Node 20 and 22

TextToSpeechStream holds only response.body, not the Response. Node 20+ fetch (undici) registers a finalizer that cancels the body of a garbage-collected Response while that body is still unlocked and unread. So if anything async runs between await client.tts.stream() and the first read (prefetching the next sentences, or await player.open({ sampleRate: speech.info.sampleRate })), a GC in that window ends the iteration at once with zero bytes, and it looks like a successful empty response. Against sandbox: stream(), gc(), then iterate gives 0 bytes on Node 20.20.2 and 22.23.4, and 46,080 bytes on Node 18. Without a forced GC, await Promise.all(sentences.map((text) => client.tts.stream({ text }))) and then reading in order gave 4 of 50 empty streams on Node 22 and 1 of 50 on Node 20. The README pattern (iterate right away) is safe, which is why the tests do not see it.

Suggested fix: lock the body as soon as the headers arrive (a locked stream is never cancelled by that finalizer) and hand out a stream that pulls from that reader. With this, the 31 TTS tests pass, both repros return the full audio, and a cut body still throws the retryable error.

const reader = response.body.getReader();
const body = new ReadableStream<Uint8Array>({
  async pull(controller) {
    const { done, value } = await reader.read();
    if (done) controller.close();
    else controller.enqueue(value);
  },
  cancel: (reason) => reader.cancel(reason),
});
return new TextToSpeechStream(speechResponseInfo(response.headers, request.voice_id), body);

[aleks-mitov]

Major: stream() stops honoring options.signal once the headers arrive

settled() runs in the finally right after fetch resolves (line 276). Besides clearing the timer, it removes the listener that forwards the caller's signal, so aborting options.signal during playback does nothing: the body keeps downloading and the loop keeps yielding. SpeechRequestOptions.signal is documented as "An external abort signal, combined with the timeout", plain fetch aborts the body on its signal, and synthesize() does keep forwarding it. An AbortController is the natural barge-in for an HTTP stream, so this hits voice agents directly. Repro: abort right after the first chunk, and the loop still read all 50 of 50 chunks from a local server (Node 18, 20 and 22); against sandbox it read another 219 to 225 chunks (about 850 KB) after abort().

Suggested fix: at the headers, stop only the timer; keep forwarding the caller's abort until the body ends, and surface it as the abort rather than as a retryable cut.

return {
  signal: controller.signal,
  headersArrived: () => clearTimeout(timer),
  release: () => {
    clearTimeout(timer);
    external?.removeEventListener("abort", forward);
  },
};
// stream(): headersArrived() once fetch resolves, release() if it throws or the status is not ok;
// pass options.signal and release into TextToSpeechStream.
// iterator, around reader.read(): if (this.signal?.aborted) throw this.signal.reason;
// iterator finally (and cancel()): this.release?.();

This shape stops after 1 chunk with AbortError on Node 18, 20 and 22. A unit test that aborts after the first chunk would pin it.

[aleks-mitov]

Major: TextToSpeechStream.cancel() does nothing while a loop is reading

The iterator takes this.body.getReader() (line 156), which locks the body. this.body.cancel() on a locked stream rejects with TypeError: Invalid state: ReadableStream is locked, and .catch(() => undefined) hides it. So a barge-in handler that calls speech.cancel() from a VAD callback while the playback loop iterates gets a resolved promise, and the rest of the audio still plays. Against sandbox: cancel() from a timer at 100 ms resolved, then the loop read all 227 chunks (871,680 bytes). Today only break inside the loop releases the connection, and that is the only path the tests cover.

Suggested fix: cancel through the active reader. reader.cancel() resolves the pending read with done, so the loop ends and the connection is released.

private reader?: ReadableStreamDefaultReader<Uint8Array>;

// in [Symbol.asyncIterator]: const reader = (this.reader = this.body.getReader());
// in its finally: if (this.reader === reader) this.reader = undefined;

async cancel(): Promise<void> {
  await (this.reader ?? this.body).cancel().catch(() => undefined);
}

Plus a test that calls cancel() from outside a running loop.

[aleks-mitov]

Major: audio() picks its request at the first next(), not when it is called

audio() is an async generator, so none of its body runs until the first next(), including this line. The doc defines audio() with no id as "the request being written now", but an iterator created before generate()/flush() and iterated afterwards binds to the next request: it discards all audio and the FlushDone of the request just flushed, then waits until the session ends. Against sandbox: const it = session.audio(); session.generate("Hello from a reader created first."); session.flush(); and then iterating it timed out, with all 115,200 bytes of flush 0 dropped. Typical triggers are playAll(session.audio()) where playAll awaits a player before its loop, or Readable.from(session.audio()). The README IIFE works only because it calls next() synchronously. The same laziness means audio(99) does not throw until the first next().

Suggested fix: resolve the id when audio() is called and keep the generator for reading.

audio(flushId?: number): AsyncGenerator<Uint8Array, void, undefined> {
  const id = flushId ?? this.nextFlushId;
  if (id > this.nextFlushId) throw new RangeError(`flush_id ${id} has not been flushed yet`);
  return this.readAudio(id);
}

private async *readAudio(flushId: number): AsyncGenerator<Uint8Array, void, undefined> {
  // the current body, starting at checkPullable()
}

Keeping checkPullable() and reading = true inside the generator keeps the two tests that expect next() to reject green, and an iterator that is never iterated cannot leave reading stuck. Add a test that creates the iterator, then calls generate() and flush(), then iterates.

[aleks-mitov]

Major: Reconnecting from the close listener makes a pending audio() end normally

When the connection drops, onclose sets this.error, end() hands the closed item to the waiting reader (which resumes in a later microtask), and then emitClose() runs the close listener synchronously. If that listener reconnects (the obvious auto-reconnect pattern, since the docs say the same object can connect() again), reset() clears this.error and replaces the queue before the reader resumes. The reader then sees this.error undefined here and returns, or returns at line 571 (this.queue !== connection). The cut request looks complete, so the app never re-sends the text. Local repro (server closes 3005 with no Error frame while audio(0) waits): without a reconnect the reader rejects with closeCode: 3005, retryable: true; with s.on("close", (code) => { if (code !== 1000) s.connect().catch(() => {}); }) it resolves with no audio and no error.

Suggested fix: carry each connection's outcome on its own queue rather than the shared this.error. Enqueue { kind: "closed", error: this.error } in end(); in audio() throw item.error (marking it surfaced only when item.error === this.error); replace the early return at line 571 with draining connection until its own closed item; and make the loop-top exit at line 575 read a per-connection ended flag and error instead of this.stopped and this.error.

[aleks-mitov]

Nit: A stream() timeout and a synthesize() timeout reject with different shapes

stream() aborts with new Error("timeout") (name "Error"), while synthesize() goes through AbortSignal.timeout and rejects with a DOMException named TimeoutError, so error.name === "TimeoutError" works for one and not the other (local server that never answers, { timeout: 500 }, same on Node 18, 20 and 22). Suggested, or wrap both in a retryable TextToSpeechError:

() => controller.abort(new DOMException(`no audio within ${timeoutMs} ms`, "TimeoutError")),

[aleks-mitov]

Nit: connect() throws synchronously for a bad websocketBaseUrl

connect() is not async and calls connectionUrl() outside the Promise, so a URL that is invalid or not ws/wss throws synchronously, and session.connect().catch(handle) (for example in a close listener) does not catch it. It has also run releaseSocket() and reset() by then. The "already connected" case returns Promise.reject, so the method is inconsistent with itself. Suggested: build the URL first and turn a failure into a rejection, before releaseSocket()/reset():

let url: URL;
try {
  url = this.connectionUrl();
} catch (error) {
  return Promise.reject(error);
}

[aleks-mitov]

Nit: Generic Speech* names at the package root

SpeechRequest, SpeechRequestOptions, SpeechResponse and SpeechResponseInfo are exported from the root of a mostly speech-to-text SDK, next to SpeechModel (an STT model) and SpeechUnderstandingRequest/SpeechUnderstandingResponse. The rest of this file uses TextToSpeech*/Tts*, other products prefix their types (SyncTranscript*, Dictation*, LlmGateway*), and Python uses TTSConfig/TTSResponse/TTSResponseInfo. Renaming after release is a breaking change, so now is the cheap moment: TextToSpeechRequest, TextToSpeechRequestOptions, TextToSpeechResponse, TextToSpeechResponseInfo.

[aleks-mitov]

Nit: Package descriptions still say transcription only; this bullet packs four APIs

The PR adds a product but leaves the one-line descriptions as they were: package.json description ("supports async and real-time transcription") and keywords, README.md line 16 and CLAUDE.md line 3 ("Speech-to-text and audio intelligence SDK"). This Key APIs entry is a single 600-character bullet covering synthesize, stream, session and the error type, while every other entry is one API per bullet. Suggested: mention text-to-speech in those descriptions (and add a "text-to-speech" npm keyword), and split this into bullets for client.tts.synthesize, client.tts.stream, client.tts.session and client.tts.createTemporaryToken.

[aleks-mitov]

Nit: This timeout test cannot fail

jest-fetch-mock's body ignores the request signal, so an abort from a timer that was never cleared does not reach it: with clearTimeout(timer) removed from headersDeadline, all 31 TTS tests still pass. Adding this right after jest.advanceTimersByTime(50); passes on the PR and fails with that mutation:

expect((fetchMock.mock.calls[0][1]!.signal as AbortSignal).aborted).toBe(false);