Skip to content

Release @diffbot/typescript 0.6.0 - #2

Merged
shixish merged 31 commits into
mainfrom
feat/sdk-s2
Nov 18, 2026
Merged

shixish merged 31 commits into
mainfrom
feat/sdk-s2

Conversation

@shixish

@shixish shixish commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

@diffbot/typescript 0.6.0

The first release since 0.3.0. It adds a shared transport with retries, full response typing, DQL over POST, and named options on the endpoints the SDK already covers. It includes the 0.4.0 and 0.5.0 work, which was never published on its own.

pnpm add @diffbot/typescript@0.6.0   # or: npm i @diffbot/typescript@0.6.0

⚠ Breaking changes

  • DQL is sent as POST with a JSON body. dql() and dqlParallel() keep the same signature and return values. The token stays in the query string, and the query text moves into the body, which removes the URL length limit. Test mocks, proxies and mirrors that match on GET + ?query= need updating. See the CHANGELOG migration section. A GET escape hatch is still available through client.http.get(client.dqlUrl, ...).
  • A failed crawl() create now throws. If the crawler answers HTTP 200 with an errorCode body (for example 505 for a job name over 30 characters), crawl() throws APIError. It used to report JobCreated for a job that was never created.
  • Retries are on by default. 429, 5xx and connection errors get up to 3 attempts with backoff and jitter. crawlDeleteJob is never retried. Pass retry: false to opt out.
  • Long Retry-After values are no longer waited out. If the header asks for longer than retry.maxDelayMs (default 5000 ms), the call fails at once with RateLimitError (retryAfter holds the raw header) or, for a 5xx, with APIError. Raise maxDelayMs if you want to keep waiting.
  • crawlGetJob() returns undefined for a missing job. It used to return {}.
  • dql() throws ValidationError if extra repeats a named option. For example, use type: "text" rather than extra: { type: "text" }.
  • Type-level only: handlers return specific response types instead of JsonObject, and ChatMessage.role is narrowed. Runtime data is unchanged.

New

  • Account: account({ days }) and verifyToken(). verifyToken() returns false only on an auth error.
  • Crawl download: crawlDownload(client, name, { format, type, num }) returns JSON records, CSV, or the URL report. The crawl() watcher now uses it.
  • Extract options: extract() takes a fifth argument with timeout, mode, per-API typed fields, paging, maxPages, useProxy and more.
  • DQL named options: type (including "crawl" together with col), filterExclude, exportseparator, exportfile, jsonmode, nonCanonicalFacts, noDedupArticles, cluster.
  • Crawl create options: urlCrawlRegEx, urlProcessRegEx, pageProcessPattern, per-subdomain limits, useCanonical, notifyWebhook, notifyEmail. customHeaders also accepts an array.
  • NLP: nlp(client, text, { fields, ... }). The result type follows the requested fields.
  • Web search: webSearch() accepts 1–5 text values.
  • Transport: per-host base URLs (apiBase, kgBase, llmBase, nlpBase), a per-call timeout, HttpClient.delete(), raw and form bodies, repeated query params, and NotFoundError / JobError (both subclasses of APIError).
  • DQL formats: format: "jsonl" now streams entities and format: "csv" returns text. Both used to fail.

Fixes

  • dql(..., { raw: true }) no longer corrupts binary (xls/xlsx) exports.
  • The crawl URL-report CSV parser handles quoted commas.
  • Readable APIError messages for { errorCode, error } bodies.
  • A retried response's connection is released before the retry.

The full list is in CHANGELOG.md.

shixish and others added 30 commits November 12, 2026 18:07
…ability set

Phase 0 foundation from the SDK coverage plan: a decode module (src/decode.ts)
centralizing JSON.parse/body reads, and a reworked HttpClient that adds:

- Per-host base URLs (apiBase/kgBase/llmBase/nlpBase) with the existing
  per-endpoint *Url options kept as winning aliases (additive).
- Repeated query parameters (array-valued params use append, not set).
- DELETE, arbitrary request bodies (raw string/bytes/URLSearchParams) and a
  caller-overridable Content-Type, including automatic form-encoding for a
  URLSearchParams body.
- Per-call timeout override, separate from the client default.
- A default-on retry policy for 429/5xx/connection errors with backoff,
  jitter, and Retry-After support, capped at 3 attempts; disable client-wide
  with retry: false or per call with retryable: false.
- Response headers surfaced on APIError (and subclasses) for callers that
  need them (e.g. a future coverage-report id).
- NotFoundError (404) and JobError, both subclassing APIError so existing
  `catch (APIError)` call sites are unaffected.

All existing handlers (extract, ask, nlp, webSearch) now read response bodies
through the decode module instead of ad hoc JSON.parse/readBodyText calls.

README documents the new client options; mock-fetch test helper widened to
read URLSearchParams/Uint8Array bodies as text for assertions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- dql(raw: true) now reads the response as bytes (arrayBuffer) instead of
  decoding it as text and re-encoding it. The text round-trip replaced any
  invalid-UTF-8 byte with U+FFFD, which silently corrupted xls/xlsx (and
  non-UTF-8 CSV) exports. JSON responses are read as text and parsed as
  before; only the raw-bytes path changes.
- crawl()'s URL-report CSV parser no longer splits lines on every comma; a
  quote-aware line parser now honors quoted fields (and "" escapes), so a
  crawled URL containing a comma is no longer mis-split across columns.
- crawl()'s job-failure throw now raises JobError (subclassing APIError, so
  existing `catch (APIError)` sites are unaffected) instead of a bare
  APIError(500, ...), and crawlDeleteJob marks its request retryable: false
  since it's a GET-shaped mutation that must not be retried blindly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Vendors the 8 Diffbot OpenAPI specs under openapi/ (natural-language.json
converted once from Swagger 2.0, since openapi-typescript only accepts 3.x)
and adds `pnpm gen:types` (scripts/gen-types.mjs, openapi-typescript as a
devDependency) to regenerate the committed src/generated/*.ts from them.

Adds hand-written facade types in src/types/*, re-exporting the generated
schema types that exist (dql, web-search, natural-language all have full
response schemas) and a src/types/undocumented.ts allow-list for surfaces no
spec describes: the ontology document, Ask/RAG request+chunk shapes, the
Crawl/Bulk job record, and Extract's per-api object shapes (extract.json,
crawl.json and bulk.json document only examples, not response schemas, per
the coverage audit). Every type stays a structural `type` alias assignable to
JsonObject, with a compile-time guard asserting that in each generated-backed
file.

Also adds src/types/extra.ts's ExtraParams<NamedKeys>, which makes passing a
named option through a handler's `extra` escape hatch a compile-time error
instead of a silent no-op.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rloads

- extract(): per-api overloads (article/product/discussion/job/image/video/
  list/event, plus a fallback for Custom API names) return
  ExtractResponse<T> for the matching per-api object type instead of a bare
  JsonObject.
- dql()/dqlParallel(): overloads keyed on `format`/`raw` return DqlResponse
  (json), string (csv), Uint8Array (xls/xlsx/raw), or AsyncGenerator<DqlEntity>
  (jsonl — now actually streamed via the decode module's readNdjson instead of
  failing a JSON.parse, since a typed overload promising it would otherwise be
  a lie). DqlOptions.extra now rejects named-option keys at compile time via
  ExtraParams.
- crawlListJobs() returns CrawlJob[]; crawlGetJob() returns CrawlJob |
  undefined instead of {} for a missing job (an intentional, type-level-only
  breaking change — see README and typesafety-revision.md).
- webSearch() returns WebSearchResponse; entities() returns the
  Pick<NaturalLanguageResponse, "entities" | "sentiment"> it always requests.
- ask()/askJson(): ChatMessage.role narrows to "system" | "user" | "assistant";
  payload typed as ChatCompletionRequest; askJson gets an optional `parse`
  hook so a caller-supplied validator checks T instead of an unchecked cast.
- Ontology: reworked around OntologyDocument/OntologyTypeDef/TaxonomyNode,
  removing every internal `as` cast.

All of this is type-level only; no handler's request-building or response
parsing changed at runtime beyond the jsonl streaming addition and the
crawlGetJob return value noted above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…me behavior

Adds tests/types/*.test-d.ts, type-checked via `pnpm test:types` (vitest
--typecheck, picked up automatically by vitest's typecheck.include default —
these files are never executed, only type-checked): extract()'s per-api
overloads, dql()'s format overloads (including dqlParallel mapping each
query's own format to its own result type and the extra/named-option
collision becoming a compile error), crawlListJobs/crawlGetJob's CrawlJob
types, ask/askJson's role union and parse-hook narrowing, webSearch/entities'
response types, and an Ontology/OntologyDocument satisfies fixture.

Also covers the runtime behavior added alongside the dql/crawl/ask retyping
that didn't yet have a test: dql(format: "jsonl") actually streaming entities
instead of failing a JSON.parse, dql(format: "csv") returning text without
attempting to parse it as JSON, crawlGetJob/crawlListJobs against empty and
populated job lists, and askJson's parse hook (including its rejection
path).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds eslint.config.js scoped to src/** (tests, scripts, generated and
vendored files are excluded, not relaxed) running
tseslint.configs.strictTypeChecked, plus `pnpm lint`.

`as` stays confined to src/decode.ts per the plan: a new unsafeCast<T>()
there replaces every other module's raw `as` (auth.ts's two runtime-shape
casts, web-search.ts's snippetLength narrowing, askJson's unchecked
fallback), and src/node/** gets its own narrow exemption since Node error
narrowing (`error as NodeJS.ErrnoException`) is idiomatic there and outside
the fetch/JSON path the rule is about.

The rest of the fixes are mechanical: numbers/possibly-undefined values
through template literals need `.toString()` or a nullish default first,
`while (true)` becomes `for (;;)`, a few redundant String()/Number() calls
and non-null assertions are replaced with real narrowing, and dqlParallel's
results array gets an explicit type argument instead of inferring `any[]`
from `new Array(n)`. A handful of validation functions that exist
specifically to catch inputs violating their own parameter types (e.g.
jsonSchemaFormat's `schema === null` check, for a JS caller not using
TypeScript) keep a scoped eslint-disable with that rationale instead of
being rewritten to "prove" something the type system can't guarantee.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ase 0

Wires pnpm lint and pnpm test:types into the existing test workflow so the
new gates stay enforced going forward, and adds CHANGELOG.md documenting
what Phase 0 added, fixed, and changed at the type level (including the two
behavior changes: dql raw-mode binary correctness and crawlGetJob returning
undefined for a missing job).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PHASE0_LEFTOVERS.md: no standalone endpoint-registry data structure (the
per-host base URL resolution it would have fed landed directly in
client.ts; a real registry has no second consumer until Phase 1 adds new
operations), and response headers are surfaced on the error path only (via
APIError.headers) — the success-path opt-in the plan motivates with DQL
coverage reports is deferred to Phase 6's dqlReport, which doesn't exist yet
to shape it against.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dql()/dqlParallel() now POST {dqlUrl}?token=... with a JSON body; the public
contract (names, options, defaults, overloads) is unchanged. Adds POST/retry
tests, overload-unchanged and no-enhance/bulk/custom-export type tests, and
drops a stale phase reference.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Separate workflow, never triggered by pull requests, concurrency group of 1.
Extends the live suite with DQL POST (hit count, csv, jsonl, >8 KB query) and
entities. Default pnpm test still makes no network calls.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…0 in the changelog

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An unread body keeps the connection pinned in undici until GC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crawler reports some create failures (e.g. errorCode 505, a job name over
30 chars) as HTTP 200. crawl() used to yield JobCreated for a job that does not
exist. APIError.apiMessage now also falls back to the body's error field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…reuses it

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, per-API params)

Caller-set wire timeout with the transport timeout kept at least timeout+10s,
named options typed per API from the vendored spec, proxy/proxyAuth via typed
extra. Existing positional args and defaults are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…portfile, jsonmode, nonCanonicalFacts, noDedupArticles, cluster

Named fields on DqlOptions, sent in the POST JSON body. A named key repeated in
extra now throws ValidationError. report/view stay reachable via extra only.
Contract test covers DQL.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds urlCrawlRegEx, urlProcessRegEx, pageProcessPattern, per-subdomain limits,
useCanonical, notifyWebhook, notifyEmail as query params on the existing GET
create. customHeaders accepts string | readonly string[], sent as repeated params.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Single-document call with fields (response narrowed via Pick), lang, format,
documentType and customSummary. README snippet and example test included.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A single string is unchanged. Arrays of 1-5 are sent as repeated text= params on
the existing GET; 0 or more than 5 throws ValidationError before any request.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… re-listing each key

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 429/5xx Retry-After was honored verbatim, so `Retry-After: 3600` made one
call sleep an hour; maxDelayMs capped only the exponential backoff. Now a
Retry-After (seconds or HTTP-date) above maxDelayMs means no sleep and no
retry: the response is returned as-is, so the caller raises RateLimitError
with retryAfter populated (or APIError with the header for a 5xx). Values at
or under the cap are honored as before. Matches the Python transport.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump package.json and src/constants.ts VERSION (the user agent) from 0.3.0
to 0.6.0. pnpm-lock.yaml does not record the root package's version, so a
refresh leaves it unchanged. Add a test that fails if VERSION and
package.json disagree, so the user agent cannot drift at the next release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.4.0 and 0.5.0 were never published, so the changelog now has a single
0.6.0 entry (Keep a Changelog) covering everything since 0.3.0, with the
breaking changes listed first: DQL over POST, crawl create HTTP 200 +
errorCode now throws, Retry-After over maxDelayMs is not retried,
crawlGetJob returns undefined, dql extra collisions throw, and retries are
on by default.

PHASE0/S1/S2_LEFTOVERS.md recorded only closed items, deliberate
narrowings, and live checks that have now run. The narrowings are under
"Known limitations" in the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ub Release)

Pushing a v* tag checks that the tag, package.json version and src/constants.ts
VERSION all match and that CHANGELOG.md has a section for the version. It then
runs the tests.yml gates (Node 18/20/22 plus the worker fixture) and publishes
with `npm publish --access public --provenance` via OIDC trusted publishing. If
no trusted publisher is configured, it falls back to the NPM_TOKEN secret.
Finally it creates the GitHub Release from the matching CHANGELOG section.
workflow_dispatch runs every gate plus `npm publish --dry-run` and publishes
nothing. Live tests are not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shixish
shixish merged commit 4a081a8 into main Nov 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant