Repository navigation
Release @diffbot/typescript 0.6.0 - #2
Merged
Merged
Conversation
…ability set Phase 0 foundation from the SDK coverage plan: a decode module (src/decode.ts) centralizing JSON.parse/body reads, and a reworked HttpClient that adds: - Per-host base URLs (apiBase/kgBase/llmBase/nlpBase) with the existing per-endpoint *Url options kept as winning aliases (additive). - Repeated query parameters (array-valued params use append, not set). - DELETE, arbitrary request bodies (raw string/bytes/URLSearchParams) and a caller-overridable Content-Type, including automatic form-encoding for a URLSearchParams body. - Per-call timeout override, separate from the client default. - A default-on retry policy for 429/5xx/connection errors with backoff, jitter, and Retry-After support, capped at 3 attempts; disable client-wide with retry: false or per call with retryable: false. - Response headers surfaced on APIError (and subclasses) for callers that need them (e.g. a future coverage-report id). - NotFoundError (404) and JobError, both subclassing APIError so existing `catch (APIError)` call sites are unaffected. All existing handlers (extract, ask, nlp, webSearch) now read response bodies through the decode module instead of ad hoc JSON.parse/readBodyText calls. README documents the new client options; mock-fetch test helper widened to read URLSearchParams/Uint8Array bodies as text for assertions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- dql(raw: true) now reads the response as bytes (arrayBuffer) instead of decoding it as text and re-encoding it. The text round-trip replaced any invalid-UTF-8 byte with U+FFFD, which silently corrupted xls/xlsx (and non-UTF-8 CSV) exports. JSON responses are read as text and parsed as before; only the raw-bytes path changes. - crawl()'s URL-report CSV parser no longer splits lines on every comma; a quote-aware line parser now honors quoted fields (and "" escapes), so a crawled URL containing a comma is no longer mis-split across columns. - crawl()'s job-failure throw now raises JobError (subclassing APIError, so existing `catch (APIError)` sites are unaffected) instead of a bare APIError(500, ...), and crawlDeleteJob marks its request retryable: false since it's a GET-shaped mutation that must not be retried blindly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Vendors the 8 Diffbot OpenAPI specs under openapi/ (natural-language.json converted once from Swagger 2.0, since openapi-typescript only accepts 3.x) and adds `pnpm gen:types` (scripts/gen-types.mjs, openapi-typescript as a devDependency) to regenerate the committed src/generated/*.ts from them. Adds hand-written facade types in src/types/*, re-exporting the generated schema types that exist (dql, web-search, natural-language all have full response schemas) and a src/types/undocumented.ts allow-list for surfaces no spec describes: the ontology document, Ask/RAG request+chunk shapes, the Crawl/Bulk job record, and Extract's per-api object shapes (extract.json, crawl.json and bulk.json document only examples, not response schemas, per the coverage audit). Every type stays a structural `type` alias assignable to JsonObject, with a compile-time guard asserting that in each generated-backed file. Also adds src/types/extra.ts's ExtraParams<NamedKeys>, which makes passing a named option through a handler's `extra` escape hatch a compile-time error instead of a silent no-op. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rloads
- extract(): per-api overloads (article/product/discussion/job/image/video/
list/event, plus a fallback for Custom API names) return
ExtractResponse<T> for the matching per-api object type instead of a bare
JsonObject.
- dql()/dqlParallel(): overloads keyed on `format`/`raw` return DqlResponse
(json), string (csv), Uint8Array (xls/xlsx/raw), or AsyncGenerator<DqlEntity>
(jsonl — now actually streamed via the decode module's readNdjson instead of
failing a JSON.parse, since a typed overload promising it would otherwise be
a lie). DqlOptions.extra now rejects named-option keys at compile time via
ExtraParams.
- crawlListJobs() returns CrawlJob[]; crawlGetJob() returns CrawlJob |
undefined instead of {} for a missing job (an intentional, type-level-only
breaking change — see README and typesafety-revision.md).
- webSearch() returns WebSearchResponse; entities() returns the
Pick<NaturalLanguageResponse, "entities" | "sentiment"> it always requests.
- ask()/askJson(): ChatMessage.role narrows to "system" | "user" | "assistant";
payload typed as ChatCompletionRequest; askJson gets an optional `parse`
hook so a caller-supplied validator checks T instead of an unchecked cast.
- Ontology: reworked around OntologyDocument/OntologyTypeDef/TaxonomyNode,
removing every internal `as` cast.
All of this is type-level only; no handler's request-building or response
parsing changed at runtime beyond the jsonl streaming addition and the
crawlGetJob return value noted above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…me behavior Adds tests/types/*.test-d.ts, type-checked via `pnpm test:types` (vitest --typecheck, picked up automatically by vitest's typecheck.include default — these files are never executed, only type-checked): extract()'s per-api overloads, dql()'s format overloads (including dqlParallel mapping each query's own format to its own result type and the extra/named-option collision becoming a compile error), crawlListJobs/crawlGetJob's CrawlJob types, ask/askJson's role union and parse-hook narrowing, webSearch/entities' response types, and an Ontology/OntologyDocument satisfies fixture. Also covers the runtime behavior added alongside the dql/crawl/ask retyping that didn't yet have a test: dql(format: "jsonl") actually streaming entities instead of failing a JSON.parse, dql(format: "csv") returning text without attempting to parse it as JSON, crawlGetJob/crawlListJobs against empty and populated job lists, and askJson's parse hook (including its rejection path). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds eslint.config.js scoped to src/** (tests, scripts, generated and vendored files are excluded, not relaxed) running tseslint.configs.strictTypeChecked, plus `pnpm lint`. `as` stays confined to src/decode.ts per the plan: a new unsafeCast<T>() there replaces every other module's raw `as` (auth.ts's two runtime-shape casts, web-search.ts's snippetLength narrowing, askJson's unchecked fallback), and src/node/** gets its own narrow exemption since Node error narrowing (`error as NodeJS.ErrnoException`) is idiomatic there and outside the fetch/JSON path the rule is about. The rest of the fixes are mechanical: numbers/possibly-undefined values through template literals need `.toString()` or a nullish default first, `while (true)` becomes `for (;;)`, a few redundant String()/Number() calls and non-null assertions are replaced with real narrowing, and dqlParallel's results array gets an explicit type argument instead of inferring `any[]` from `new Array(n)`. A handful of validation functions that exist specifically to catch inputs violating their own parameter types (e.g. jsonSchemaFormat's `schema === null` check, for a JS caller not using TypeScript) keep a scoped eslint-disable with that rationale instead of being rewritten to "prove" something the type system can't guarantee. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ase 0 Wires pnpm lint and pnpm test:types into the existing test workflow so the new gates stay enforced going forward, and adds CHANGELOG.md documenting what Phase 0 added, fixed, and changed at the type level (including the two behavior changes: dql raw-mode binary correctness and crawlGetJob returning undefined for a missing job). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
PHASE0_LEFTOVERS.md: no standalone endpoint-registry data structure (the per-host base URL resolution it would have fed landed directly in client.ts; a real registry has no second consumer until Phase 1 adds new operations), and response headers are surfaced on the error path only (via APIError.headers) — the success-path opt-in the plan motivates with DQL coverage reports is deferred to Phase 6's dqlReport, which doesn't exist yet to shape it against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dql()/dqlParallel() now POST {dqlUrl}?token=... with a JSON body; the public
contract (names, options, defaults, overloads) is unchanged. Adds POST/retry
tests, overload-unchanged and no-enhance/bulk/custom-export type tests, and
drops a stale phase reference.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Separate workflow, never triggered by pull requests, concurrency group of 1. Extends the live suite with DQL POST (hit count, csv, jsonl, >8 KB query) and entities. Default pnpm test still makes no network calls. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…0 in the changelog Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An unread body keeps the connection pinned in undici until GC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The crawler reports some create failures (e.g. errorCode 505, a job name over 30 chars) as HTTP 200. crawl() used to yield JobCreated for a job that does not exist. APIError.apiMessage now also falls back to the body's error field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…reuses it Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, per-API params) Caller-set wire timeout with the transport timeout kept at least timeout+10s, named options typed per API from the vendored spec, proxy/proxyAuth via typed extra. Existing positional args and defaults are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…portfile, jsonmode, nonCanonicalFacts, noDedupArticles, cluster Named fields on DqlOptions, sent in the POST JSON body. A named key repeated in extra now throws ValidationError. report/view stay reachable via extra only. Contract test covers DQL. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds urlCrawlRegEx, urlProcessRegEx, pageProcessPattern, per-subdomain limits, useCanonical, notifyWebhook, notifyEmail as query params on the existing GET create. customHeaders accepts string | readonly string[], sent as repeated params. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Single-document call with fields (response narrowed via Pick), lang, format, documentType and customSummary. README snippet and example test included. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A single string is unchanged. Arrays of 1-5 are sent as repeated text= params on the existing GET; 0 or more than 5 throws ValidationError before any request. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… re-listing each key Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 429/5xx Retry-After was honored verbatim, so `Retry-After: 3600` made one call sleep an hour; maxDelayMs capped only the exponential backoff. Now a Retry-After (seconds or HTTP-date) above maxDelayMs means no sleep and no retry: the response is returned as-is, so the caller raises RateLimitError with retryAfter populated (or APIError with the header for a 5xx). Values at or under the cap are honored as before. Matches the Python transport. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump package.json and src/constants.ts VERSION (the user agent) from 0.3.0 to 0.6.0. pnpm-lock.yaml does not record the root package's version, so a refresh leaves it unchanged. Add a test that fails if VERSION and package.json disagree, so the user agent cannot drift at the next release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0.4.0 and 0.5.0 were never published, so the changelog now has a single 0.6.0 entry (Keep a Changelog) covering everything since 0.3.0, with the breaking changes listed first: DQL over POST, crawl create HTTP 200 + errorCode now throws, Retry-After over maxDelayMs is not retried, crawlGetJob returns undefined, dql extra collisions throw, and retries are on by default. PHASE0/S1/S2_LEFTOVERS.md recorded only closed items, deliberate narrowings, and live checks that have now run. The narrowings are under "Known limitations" in the changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ub Release) Pushing a v* tag checks that the tag, package.json version and src/constants.ts VERSION all match and that CHANGELOG.md has a section for the version. It then runs the tests.yml gates (Node 18/20/22 plus the worker fixture) and publishes with `npm publish --access public --provenance` via OIDC trusted publishing. If no trusted publisher is configured, it falls back to the NPM_TOKEN secret. Finally it creates the GitHub Release from the matching CHANGELOG section. workflow_dispatch runs every gate plus `npm publish --dry-run` and publishes nothing. Live tests are not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@diffbot/typescript 0.6.0
The first release since 0.3.0. It adds a shared transport with retries, full response typing, DQL over POST, and named options on the endpoints the SDK already covers. It includes the 0.4.0 and 0.5.0 work, which was never published on its own.
pnpm add @diffbot/typescript@0.6.0 # or: npm i @diffbot/typescript@0.6.0⚠ Breaking changes
POSTwith a JSON body.dql()anddqlParallel()keep the same signature and return values. The token stays in the query string, and the query text moves into the body, which removes the URL length limit. Test mocks, proxies and mirrors that match onGET+?query=need updating. See the CHANGELOG migration section. A GET escape hatch is still available throughclient.http.get(client.dqlUrl, ...).crawl()create now throws. If the crawler answers HTTP 200 with anerrorCodebody (for example505for a job name over 30 characters),crawl()throwsAPIError. It used to reportJobCreatedfor a job that was never created.429,5xxand connection errors get up to 3 attempts with backoff and jitter.crawlDeleteJobis never retried. Passretry: falseto opt out.Retry-Aftervalues are no longer waited out. If the header asks for longer thanretry.maxDelayMs(default 5000 ms), the call fails at once withRateLimitError(retryAfterholds the raw header) or, for a 5xx, withAPIError. RaisemaxDelayMsif you want to keep waiting.crawlGetJob()returnsundefinedfor a missing job. It used to return{}.dql()throwsValidationErrorifextrarepeats a named option. For example, usetype: "text"rather thanextra: { type: "text" }.JsonObject, andChatMessage.roleis narrowed. Runtime data is unchanged.New
account({ days })andverifyToken().verifyToken()returnsfalseonly on an auth error.crawlDownload(client, name, { format, type, num })returns JSON records, CSV, or the URL report. Thecrawl()watcher now uses it.extract()takes a fifth argument withtimeout,mode, per-API typedfields,paging,maxPages,useProxyand more.type(including"crawl"together withcol),filterExclude,exportseparator,exportfile,jsonmode,nonCanonicalFacts,noDedupArticles,cluster.urlCrawlRegEx,urlProcessRegEx,pageProcessPattern, per-subdomain limits,useCanonical,notifyWebhook,notifyEmail.customHeadersalso accepts an array.nlp(client, text, { fields, ... }). The result type follows the requestedfields.webSearch()accepts 1–5textvalues.apiBase,kgBase,llmBase,nlpBase), a per-calltimeout,HttpClient.delete(), raw and form bodies, repeated query params, andNotFoundError/JobError(both subclasses ofAPIError).format: "jsonl"now streams entities andformat: "csv"returns text. Both used to fail.Fixes
dql(..., { raw: true })no longer corrupts binary (xls/xlsx) exports.APIErrormessages for{ errorCode, error }bodies.The full list is in CHANGELOG.md.