Skip to content

Release diffbot 3.4.0 - #10

Merged
shixish merged 32 commits into
mainfrom
feat/sdk-s2
Nov 18, 2026
Merged

shixish merged 32 commits into
mainfrom
feat/sdk-s2

Conversation

@shixish

@shixish shixish commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

diffbot 3.4.0

The first release since 3.1.1. It adds a shared transport with retries, response TypedDicts with a pyright gate, DQL over POST, nlp(), and named options on the endpoints the SDK already covers. It includes the 3.2.0 and 3.3.0 work, which was never published on its own.

pip install -U "diffbot==3.4.0"   # or: uv add "diffbot>=3.4.0,<4"

The self-contained db binaries for Linux (x86_64 and aarch64) and macOS (Apple Silicon) are attached to this release.

⚠ Breaking changes

  • DQL is sent as POST with a JSON body. dql(), dql_parallel() and their async versions keep the same signature and return values. The token stays in the query string, and the query text moves into the body, which removes the URL length limit. Test mocks, proxies and mirrors that match on GET + ?query= need updating. See the CHANGELOG migration section. A GET escape hatch is still available: db.request("GET", db.dql_url, params=...).
  • A failed crawl() create now raises. If the crawler answers HTTP 200 with errorCode/error (for example errorCode 505 for a job name over 30 characters), crawl() raises APIError. It used to yield JOB_CREATED for a job that was never created.
  • Retries are on by default. 429, 5xx and connection errors get up to 3 attempts with backoff and jitter. crawl_delete_job is never retried. Pass retry=False to opt out.
  • Long Retry-After values are no longer waited out. If the header asks for longer than RetryPolicy.max_delay (default 5 s), the call raises at once with RateLimitError (retry_after holds the raw header) or, for a 5xx, with APIError. This applies to sync, async and streamed calls. Raise max_delay if you want to keep waiting.
  • web_search(snippet_length=...) is validated. Values other than short, medium, long or full raise ValidationError.
  • dql() raises ValidationError if extra repeats one of the new named options.
  • Type-level only: dql, extract, web_search, entities and the crawl_* getters return TypedDicts. Runtime values are unchanged. Use typing.cast(Dict[str, Any], ...) if a type checker objects.

New

  • Account: account(days=...) and verify_token(), sync and async.
  • Crawl download: crawl_download(name, format=..., type=..., num=...), sync and async. A JSON body for an unknown job raises JobError.
  • Extract options: extract() keyword options for timeout, mode, per-API fields, paging, max_pages, use_proxy and more, plus extra.
  • DQL named options: type (including "crawl" together with col), filter_exclude, exportseparator, exportfile, jsonmode, non_canonical_facts, no_dedup_articles, cluster.
  • Crawl create options: url_crawl_regex, url_process_regex, page_process_pattern, per-subdomain limits, use_canonical, notify_webhook, notify_email. custom_headers also accepts a list.
  • NLP: nlp(text, fields=[...], ...), sync and async.
  • Web search: web_search() accepts 1–5 text values.
  • Ontology: dql_fetch_ontology_text(), and dql_refresh_ontology on DiffbotAsync.
  • Transport: per-host base URLs (api_base, kg_base, llm_base, nlp_base), client.request(...) for any method or body with a per-call timeout, NotFoundError / JobError (both subclasses of APIError), and APIError.headers.
  • Typing: response TypedDicts exported from diffbot. They use the stdlib only and work on Python 3.9.

Fixes

  • dql(format="csv") and dql(format="jsonl") no longer crash: they return str and an iterator of entities.
  • A non-2xx response while streaming ask()/ask_json() raises AuthError/RateLimitError instead of httpx.ResponseNotRead.
  • APIError.message falls back to the error field of {errorCode, error} bodies.
  • Retry-After: inf no longer crashes time.sleep.

The full list is in CHANGELOG.md.

shixish and others added 30 commits November 13, 2026 16:09
…ility set, retry policy, and error hierarchy

Mirrors the TypeScript SDK's Phase 0 transport work (same branch) in Python idiom:

- Shared sync/async request-shaping and response-handling helpers in a new
  _transport.py, replacing the duplicated Diffbot/DiffbotAsync._raise_for_status
  methods and the repeated client._http.get(...) + raise pattern across every
  handler module with a single client.request(method, url, ...) used by both.
- Per-host base URLs (api_base/kg_base/llm_base/nlp_base) with the existing
  per-endpoint *_url constructor options kept as winning aliases (additive).
  No standalone "endpoint registry" data structure — Phase 0 adds zero new
  endpoints, so the only thing it would serve today is base-URL resolution
  (done directly in client.py) and a retryable flag (done ad hoc), matching
  the TypeScript SDK's same call on this branch (see its PHASE0_LEFTOVERS.md).
- Repeated query parameters (declared via a Sequence-typed QueryParams; httpx
  already encodes list-valued params as repeated keys).
- DELETE + arbitrary method/body/Content-Type, form-encoded (data=) and raw
  (content=) request bodies, per-call timeout override, and a follow_redirects
  override, all threaded through client.request().
- A default-on retry policy (RetryPolicy dataclass) for 429/5xx/connection
  errors with backoff, jitter, and Retry-After support, capped at 3 attempts;
  disable client-wide with retry=False or per call with retryable=False.
  crawl_delete_job/_async now pass retryable=False since it's a GET-shaped
  mutation that must not be retried blindly.
- Response headers surfaced on APIError (and subclasses) via a new headers
  field, populated from the error response.
- NotFoundError (404) and JobError, both subclassing APIError so existing
  `except APIError` call sites are unaffected. crawl's job-failure path now
  raises JobError instead of a synthesized APIError(500, ...).
- fix(ask): read the response body before raising on a non-2xx streaming
  response (sync and async). Previously client._raise_for_status accessed
  response.text immediately after opening the stream, which raises
  httpx.ResponseNotRead instead of AuthError/RateLimitError once the
  connection is genuinely still streaming (tests use a real byte-stream mock
  transport, since httpx.MockTransport's content= pre-fills the body and masks
  this). ask()/ask_json() now go through client.request(..., stream=True),
  which also means they pick up the retry policy like every other handler.
- web_search.py's sync/async bodies, previously 100% duplicated, now share a
  single _build_web_search_request() helper.

Also adds additive TypedDict/Literal type aliases (types.py) for existing
surfaces only — ChatMessage, ResponseFormat, ExportFormat, SnippetLength — used
in ask()/dql()/web_search() signatures. No Pydantic, no runtime behavior
change; Python floor stays >=3.9 (stdlib typing only).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Client configuration table and adds examples for api_base/kg_base/
llm_base/nlp_base (pointing the whole SDK at a staging host with one option)
and retry/RetryPolicy, matching the TypeScript SDK's README for the same Phase
0 work. Adds the matching tests per AGENTS.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… changelog entry

PHASE0_LEFTOVERS.md: no standalone endpoint-registry data structure (same call
as the TypeScript SDK on this branch — Phase 0 adds zero new endpoints, so a
registry has no second consumer yet), response headers surfaced on the error
path only (success-path headers deferred to Phase 6's dql_report, which
doesn't exist yet to shape it against), confirmation that Python's dql(raw=True)
never had the TypeScript SDK's binary-corruption bug, the ontology async/text
asymmetry being Phase 1 scope rather than Phase 0, and the scope of the
TypedDict/Literal additions (existing request parameters, not response shapes
— this repo has no type-checking CI gate to shape response overlays against).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dql()/dql_parallel() and async twins now POST to the DQL endpoint with a JSON
body; the token stays in the query string. Public contract unchanged. extra
widens to Mapping[str, Any].

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Separate workflow, never on pull requests, concurrency of 1. Adds a cheap
live DQL POST/entities suite asserting on shape only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on note

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Several Diffbot APIs answer with {errorCode, error} rather than {message}.
Mirror the TypeScript SDK's tryParseErrorBody so APIError.message (and str(err))
carry that text, and ignore non-string message/requestId values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocstring

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erify_token, crawl_download, extract options

- types.py: stdlib TypedDict(total=False) response shapes (DqlResponse,
  WebSearchResponse, EntitiesResponse, ExtractResponse + per-API, CrawlJob,
  AccountResponse, AskChunk) and per-API extract option/field Literals.
- dql: @Overloads by format/raw; csv -> str, jsonl -> iterator of entities
  (previously both crashed in response.json() without raw=True).
- account(days=None) / verify_token() (GET /v4/account); only AuthError -> False.
- crawl_download(name, format, type, num); the crawl watcher reuses it.
- extract: keyword-only timeout/mode/fields/etc; transport timeout raised to
  at least timeout + 10s; extra for proxy/proxyAuth.
- pyright dev dependency + config (basic on src/, strict on types.py), CLI
  annotation fixes, tests/typing assert_type checks.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, crawl_download

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, account/verify_token

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… type AskChunk in the SSE parser

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l create-error body

- crawl_download(type="urls") returned a JSON decode error: that report is always
  CSV, whatever format says. Return the text (matching the TypeScript SDK) and add a
  type="urls" -> str overload.
- crawl_download raised nothing for a non-list JSON body such as
  {errorCode, error} for an unknown job and handed the dict back as records.
  Raise JobError carrying the job name, as TypeScript does.
- crawl create with HTTP 200 + errorCode now raises APIError with the real status,
  body and headers instead of a synthesized body; message comes from the error
  field via the APIError fallback.
- Document the 30-character job name limit (README, crawl docstring) and add the
  create-error fix to the 3.3.0 changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xportfile, jsonmode, non_canonical_facts, no_dedup_articles, cluster

Keyword-only options on dql()/dql_async() (and so dql_parallel), sent in the POST
body. Explicit option plus the same wire key in extra raises ValidationError.
report/view stay reachable via extra only.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
url_crawl_regex, url_process_regex, page_process_pattern, max_to_crawl_per_subdomain,
max_to_process_per_subdomain, use_canonical, notify_webhook, notify_email on crawl()
(query params on the existing GET). custom_headers accepts a str or a sequence, sent
as repeated params. Recurring-crawl admin stays out.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nlp(text, *, fields, lang, format, document_type, custom_summary) plus async twin,
single document. Adds NlpField/NlpFormat/NlpCustomSummary/NlpResponse types, README
snippet and its test. entities() keeps hard-coding fields=entities,sentiment.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…arity

text may be a str or 1-5 strings (repeated text= params on the existing GET); 0 or >5
raises ValidationError. snippet_length is case-normalized and validated like the TS
SDK; invalid values now raise ValidationError. Docstrings label snippet_length/rerank
as undocumented.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Closes the Python/TypeScript ontology asymmetry (PHASE0_LEFTOVERS §4). Raw JSON text
fetch on both clients; async refresh on DiffbotAsync (dql_refresh_ontology) and in
diffbot.kg (dql_refresh_ontology_async). dql_fetch_ontology reuses the text fetch.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd async

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 429/5xx Retry-After was honored verbatim, so `Retry-After: 3600` made one
call sleep an hour; max_delay capped only the exponential backoff. Now a
Retry-After (seconds or HTTP-date) above RetryPolicy.max_delay means no sleep
and no retry: the response is raised at once as RateLimitError with
retry_after populated (or APIError with the header for a 5xx), in the sync,
async and streamed paths. Values at or under the cap are honored as before.
Non-finite seconds (inf/nan) fall back to backoff instead of crashing
time.sleep. Matches the TypeScript SDK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump pyproject.toml and the uv.lock entry for the project from 3.1.0 to
3.4.0. __version__, the db --version string, and the User-Agent all read
installed metadata, so there is no second constant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.2.0 and 3.3.0 were never published, so the changelog now has a single
3.4.0 entry (Keep a Changelog) covering everything since 3.1.x, with the
breaking changes listed first: DQL over POST, crawl create HTTP 200 +
errorCode now raises, Retry-After over max_delay is not retried, retries
are on by default, snippet_length is validated, and dql extra collisions
raise.

PHASE0/S1/S2_LEFTOVERS.md recorded only closed items, deliberate
narrowings, and live checks that have now run. The narrowings are under
"Known limitations" in the changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.1.1 is on PyPI but feat/sdk-s2 forked from 3.1.0, so releasing 3.4.0
without this would undo it: db -o now creates missing parent folders, the
verify-release targets refresh uv's index cache, and the macOS release
binary builds on macos-15. Conflicts were only the version line in
pyproject.toml and uv.lock; 3.4.0 is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shixish and others added 2 commits November 17, 2026 16:28
Add .github/workflows/publish-pypi.yml: verify (tag == pyproject version ==
diffbot.__version__, unit tests, pyright), build (uv build sdist + wheel,
twine check --strict, artifact), and publish (tags only, environment "pypi",
id-token: write, pypa/gh-action-pypi-publish@release/v1; no API token).
workflow_dispatch is a dry run that never uploads. Concurrency never cancels
an in-flight publish. release.yml (binaries) is unchanged.

Document the release path (merge to main, push tag vX.Y.Z) in the README and
mark `make release` as a manual fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Release workflow on v* tags: verify, build sdist/wheel and db binaries,
publish to PyPI via trusted publishing (after binaries build), then the
GitHub Release. Concurrency never cancels an in-flight publish.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@shixish
shixish merged commit f15d9c1 into main Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant