Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
a605945
feat(transport): Phase 0 foundation — per-host bases, full HTTP capab…
shixish Nov 14, 2026
9902bc3
docs: document per-host bases and retry policy in the README
shixish Nov 14, 2026
b022907
docs: note two deliberately deferred Phase 0 items; add an Unreleased…
shixish Nov 14, 2026
92fc982
feat(dql): send DQL as POST with a JSON body
shixish Nov 17, 2026
0d08d8e
ci: live tests on push to main, workflow_dispatch, cron 17 14 1,15 * *
shixish Nov 17, 2026
2eecaf4
docs: close Phase 0 leftovers under the reduced-scope plan
shixish Nov 17, 2026
b0ae50b
docs(changelog): name 3.2.0, add Changed section and DQL POST migrati…
shixish Nov 17, 2026
2652c89
fix(errors): fall back to the error key for APIError.message
shixish Nov 18, 2026
58ebff4
docs(kg): drop the stale entity-enhancement mention from the module d…
shixish Nov 18, 2026
a291377
docs(changelog): note the APIError.message fallback in 3.2.0
shixish Nov 18, 2026
85a9e5c
feat: response TypedDicts, dql overloads and csv/jsonl fix, account/v…
shixish Nov 17, 2026
51f6700
ci: run pyright in tests.yml; extend live suite over account, extract…
shixish Nov 17, 2026
97d53e4
docs(readme): extract options, dql formats, crawl start/poll/download…
shixish Nov 17, 2026
13cd90d
docs: 3.3.0 changelog, S1 leftovers, mark response TypedDicts landed;…
shixish Nov 17, 2026
432ce4a
fix(test): keep live crawl job name within the 30-char API limit
shixish Nov 18, 2026
653915b
fix(crawl): raise APIError when create returns HTTP 200 with errorCode
shixish Nov 18, 2026
45e7cdc
fix(crawl): crawl_download urls report and error bodies; keep the rea…
shixish Nov 18, 2026
4c7d899
feat(dql): named params type, col, filter_exclude, exportseparator, e…
shixish Nov 17, 2026
e3570af
feat(crawl): named create options and multi-value custom_headers
shixish Nov 17, 2026
b110b40
feat(nlp): nlp() on the existing NL endpoint, entities() unchanged
shixish Nov 17, 2026
4a83898
feat(web-search): accept 1-5 text values; snippet_length validation p…
shixish Nov 17, 2026
d2558f5
feat(ontology): dql_fetch_ontology_text and dql_refresh_ontology_async
shixish Nov 17, 2026
dcc9754
feat(live,docs): S2 live cases, 3.4.0 changelog, S2 leftovers
shixish Nov 17, 2026
dba418f
docs: record Q2/Q3 live confirmations in S2 leftovers
shixish Nov 18, 2026
427982d
refactor(crawl): share the named create-option mapping between sync a…
shixish Nov 18, 2026
5fe3c47
fix(transport): stop retrying when Retry-After exceeds max_delay
shixish Nov 18, 2026
1ff55a0
chore(release): 3.4.0
shixish Nov 18, 2026
7d0241b
docs: one 3.4.0 changelog entry; drop the resolved leftovers files
shixish Nov 18, 2026
22634db
Merge origin/main (3.1.1) into feat/sdk-s2
shixish Nov 18, 2026
40e98ba
docs(changelog): say 3.4.0 includes 3.1.1
shixish Nov 18, 2026
9ad9f63
ci: publish to PyPI via trusted publishing on v* tags
shixish Nov 18, 2026
b7937a8
ci: merge PyPI publishing into release.yml (no environment)
shixish Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/live-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Live tests

# Real-API tests (`pytest -m live`). Deliberately never run on pull requests,
# so a fork or PR can neither spend credits nor reach the secret.
on:
push:
branches: [main]
workflow_dispatch:
schedule:
- cron: "17 14 1,15 * *"

permissions:
contents: read

jobs:
live:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
concurrency:
group: live-tests
cancel-in-progress: false
env:
DIFFBOT_TOKEN: ${{ secrets.DIFFBOT_TOKEN }}
# The SDK's own credential lookup reads DIFFBOT_API_TOKEN.
DIFFBOT_API_TOKEN: ${{ secrets.DIFFBOT_TOKEN }}
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"

- name: Install dependencies (locked)
run: uv sync --frozen --extra dev

- name: Run live tests
run: uv run pytest -m live -v
147 changes: 121 additions & 26 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,110 @@
name: Release Binaries
name: Release

# Builds the self-contained `db` CLI binary for each supported platform and
# publishes them (with SHA256 checksums) to a GitHub Release. Triggered by
# pushing a version tag (e.g. v0.2.1); also runnable manually for dry runs.
# One workflow for a release. Triggered by pushing a version tag (e.g. v3.4.0);
# also runnable manually (workflow_dispatch) as a dry run that verifies and
# builds everything but never publishes.
#
# verify -> build-dist (sdist + wheel) --\
# verify -> build-binaries (db CLI) -----+-> publish-pypi -> github-release
#
# PyPI publishing uses trusted publishing (OIDC), so no API token is stored.
# One-time setup (maintainer): on pypi.org > diffbot > Manage > Publishing, add
# a GitHub publisher with owner `diffbot`, repository `diffbot-python`,
# workflow `release.yml`, and NO environment (leave it blank).
on:
push:
tags: ["v*"]
workflow_dispatch:

permissions:
contents: write
contents: read

# Re-pushing a tag (or re-dispatching) cancels an in-flight run for the same
# ref so a stuck build can't pile up behind a newer attempt.
# Never cancel an in-flight run: a cancelled publish could leave PyPI with only
# some of the files for a version (PyPI doesn't allow re-uploading them) or
# leave PyPI and the GitHub Release out of sync. Runs for the same ref queue.
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: false

jobs:
build:
verify:
name: Verify version, tests, and types
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"

- name: Install dependencies (locked)
run: uv sync --frozen --extra dev

# __version__ is read from the installed distribution metadata, so this
# checks that pyproject.toml's version is what the package reports.
- name: Check tag matches package version
run: |
set -euo pipefail
pkg=$(grep -m1 '^version' pyproject.toml | cut -d'"' -f2)
runtime=$(uv run --frozen python -c 'import diffbot; print(diffbot.__version__)')
if [ "$pkg" != "$runtime" ]; then
echo "::error::pyproject.toml version ${pkg} does not match diffbot.__version__ ${runtime}."
exit 1
fi
if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
tag=${GITHUB_REF_NAME#v}
if [ "$pkg" != "$tag" ]; then
echo "::error::Tag v${tag} does not match pyproject.toml version ${pkg}. Bump pyproject.toml (e.g. 'make bump-patch') and re-tag."
exit 1
fi
fi
echo "Package version: ${pkg}"

# Unit tests only: pyproject's addopts deselect `-m live` by default.
- name: Run tests
run: uv run --frozen pytest

- name: Run pyright
run: uv run --frozen pyright

build-dist:
name: Build sdist and wheel
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"

- name: Build
run: uv build --sdist --wheel --out-dir dist

- name: Check distributions
run: uvx twine check --strict dist/*

- name: Check built version matches pyproject.toml
run: |
set -euo pipefail
pkg=$(grep -m1 '^version' pyproject.toml | cut -d'"' -f2)
ls dist
test -f "dist/diffbot-${pkg}.tar.gz"
test -f "dist/diffbot-${pkg}-py3-none-any.whl"

- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: python-dist
path: dist/
if-no-files-found: error

build-binaries:
needs: verify
name: Build ${{ matrix.target_os }}-${{ matrix.arch }}
runs-on: ${{ matrix.runner }}
# Cap the wait so a capacity-starved runner fails this leg in minutes
Expand Down Expand Up @@ -46,20 +132,6 @@ jobs:
steps:
- uses: actions/checkout@v4

# Fail fast if the tag doesn't match pyproject.toml's version, so the
# GitHub Release name can never disagree with the version the binaries
# report (db --version) or the one published to PyPI.
- name: Check tag matches package version
if: startsWith(github.ref, 'refs/tags/')
run: |
pkg=$(grep -m1 '^version' pyproject.toml | cut -d'"' -f2)
tag=${GITHUB_REF_NAME#v}
if [ "$pkg" != "$tag" ]; then
echo "::error::Tag v${tag} does not match pyproject.toml version ${pkg}. Bump pyproject.toml (e.g. 'make bump-patch') and re-tag."
exit 1
fi
echo "Tag and package version agree: ${pkg}"

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
Expand All @@ -86,12 +158,35 @@ jobs:
dist/db-${{ matrix.target_os }}-${{ matrix.arch }}.sha256
if-no-files-found: error

release:
publish-pypi:
name: Publish to PyPI
# Only after the sdist/wheel AND all db binaries built successfully.
needs: [build-dist, build-binaries]
# Tag pushes only; workflow_dispatch stops after the builds.
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
runs-on: ubuntu-latest
timeout-minutes: 15
# No GitHub environment: the PyPI trusted publisher has none configured.
permissions:
id-token: write
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: python-dist
path: dist/

- name: Publish
uses: pypa/gh-action-pypi-publish@release/v1

github-release:
name: Publish GitHub Release
needs: build
needs: [build-dist, build-binaries, publish-pypi]
runs-on: ubuntu-latest
# Only publish for real tag pushes; workflow_dispatch runs just build + verify.
if: startsWith(github.ref, 'refs/tags/')
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
permissions:
contents: write
steps:
- name: Download all artifacts
uses: actions/download-artifact@v4
Expand Down
18 changes: 18 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,24 @@ jobs:
- name: Run tests
run: uv run pytest

# Static typing gate: pyright (basic on src/, strict on types.py) plus the
# assert_type checks in tests/typing. One Python version is enough.
typecheck:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install uv
uses: astral-sh/setup-uv@v5
with:
python-version: "3.12"

- name: Install dependencies (locked)
run: uv sync --frozen --extra dev

- name: Run pyright
run: uv run pyright

# Catch breakage against the lowest dependency versions our ranges allow,
# which the pinned lockfile would otherwise hide.
test-lowest:
Expand Down
154 changes: 154 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
# Changelog

All notable changes to the `diffbot` Python package are documented here. The format
follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
[Semantic Versioning](https://semver.org/).

## [3.4.0] - 2026-11-17

First release since 3.1.1 (and includes everything in it). It rolls up three development lines that were never
published on their own: the shared transport, sync/async de-duplication and DQL over
POST (planned as 3.2.0), response types plus `account`/`verify_token`/`crawl_download`/
extract options (3.3.0), and named options on existing endpoints plus `nlp()` (3.4.0).
No new Diffbot API areas are added.

### ⚠ Breaking changes

- **DQL is sent as `POST` with a JSON body.** `dql()`, `dql_parallel()` and their async
twins keep the same signature and results, but mocks, proxies, or mirrors keyed on
`GET` + `?query=` stop matching. See [Migrating DQL](#migrating-dql-mocks-and-proxies)
below.
- **`crawl()` raises `APIError` when the create call answers HTTP 200 with
`errorCode`/`error`** (for example errorCode 505, job name over 30 characters). It used
to yield `JOB_CREATED` for a job that was never created. The error keeps the real
status (200), body, and headers.
- **A `Retry-After` above `RetryPolicy.max_delay` (default 5.0 s) is no longer waited
out or retried.** The call raises at once: `RateLimitError` with `retry_after` = raw
header, or `APIError` with `headers["retry-after"]` for a 5xx (sync, async, and
streamed). Raise `max_delay` to keep waiting.
- **Requests are now retried by default** (`429`, `5xx`, connection errors; 3 attempts
with backoff and jitter). `crawl_delete_job` is never retried. Pass `retry=False` for
the old single-attempt behavior.
- **`web_search(snippet_length=...)` is validated:** anything other than `short`,
`medium`, `long`, `full` (case-insensitive) raises `ValidationError` instead of being
sent upstream.
- **`dql()` raises `ValidationError` when `extra` repeats one of the new named options**
(`type`, `col`, `filter_exclude`, ...).
- **Type-level:** `dql`, `extract`, `web_search`, `entities`, `crawl_list_jobs`,
`crawl_get_job` return `TypedDict`s instead of `Dict[str, Any]`, and several methods
gained `@overload`s. Runtime values are unchanged; a type checker may flag undeclared
keys (escape hatch: `typing.cast(Dict[str, Any], resp)`).

### Added

- **Transport:** per-host base URLs (`api_base`, `kg_base`, `llm_base`, `nlp_base` on
`Diffbot` and `DiffbotAsync`; the older `*_url` options still win when both are set);
documented repeated query params; `client.request(method, url, ...)` with `DELETE`,
form (`data=`) and raw (`content=`) bodies, a per-call `timeout`, and a
`follow_redirects` override; the `RetryPolicy` dataclass and `retry` option with
per-call `retryable=False`.
- **Errors:** `NotFoundError` (404) and `JobError`, both subclasses of `APIError`;
`APIError.headers` from the error response.
- **Types:** request aliases (`ChatMessage`, `ResponseFormat`, `ExportFormat`,
`SnippetLength`) and response `TypedDict`s exported from `diffbot` (`DqlResponse`/
`DqlEntity`, `WebSearchResponse`/`WebSearchResult`, `EntitiesResponse`,
`ExtractResponse` and per-API variants, `CrawlJob`, `CrawlDataRecord`,
`AccountResponse`, `AskChunk`, ...). stdlib only, Python 3.9-compatible, no Pydantic.
- `account(days=None)` and `verify_token()` (sync and async) over `GET /v4/account`.
`verify_token()` returns `False` only on an auth failure. The account response carries
`token` and `childTokens`; don't log it whole.
- `crawl_download(name, *, format="json", type=None, num=None)` (sync and async) over
`GET /v3/crawl/data`. `format="csv"` or `type="urls"` returns `str`, otherwise a list
of records; a non-list JSON body (e.g. unknown job) raises `JobError`. The crawl
watcher uses it. For tools that can't block: `crawl(..., watch=False)` ->
`crawl_get_job` -> `crawl_download`.
- `extract()` keyword-only options: `timeout` (server-side ms, default 30000), `mode`,
`fields` (per-API `Literal` names), `fallback`, `discussion`, `paging`, `max_pages`,
`natural_language`, `summary_num_sentences`, `max_tags`, `tag_confidence`,
`category_confidence`, `use_proxy`, and `extra` (`proxy`/`proxyAuth`; collisions with a
named option or `token`/`url` raise `ValidationError`). Setting `timeout` raises the
per-call transport timeout to at least `timeout + 10s`.
- `dql()`/`dql_async()` (and `dql_parallel`) keyword-only options `type` (`"query"`,
`"text"`, `"queryTextFallback"`, `"crawl"`), `col`, `filter_exclude`,
`exportseparator`, `exportfile`, `jsonmode`, `non_canonical_facts`,
`no_dedup_articles`, `cluster`, sent in the POST body. `report` and `view` stay
reachable via `extra`. Types: `DqlType`, `DqlJsonMode`, `DqlCluster`, `DqlOptions`.
- `crawl()` create options `url_crawl_regex`, `url_process_regex`,
`page_process_pattern`, `max_to_crawl_per_subdomain`, `max_to_process_per_subdomain`,
`use_canonical`, `notify_webhook`, `notify_email`; `custom_headers` accepts a string or
a sequence (repeated params). The 30-character job-name limit is documented.
- `nlp(text, *, fields, lang, format, document_type, custom_summary)` (sync and async) on
the existing NL endpoint, single document. `entities()` is unchanged. Types:
`NlpField`, `NlpFormat`, `NlpCustomSummary`, `NlpResponse`.
- `web_search(text=...)` accepts a string or 1-5 strings (repeated `text=`);
out-of-range lengths raise `ValidationError` before any request.
- `dql_fetch_ontology_text()` (sync and async); `dql_refresh_ontology` on
`DiffbotAsync` (module function `diffbot.kg.dql_refresh_ontology_async`).
- Dev: `pyright` gate (basic on `src/`, strict on `types.py`, plus `tests/typing`) in CI;
live test workflow (push to `main`, manual dispatch, twice-monthly cron; never on PRs).

### Changed

- `dql()` request shape: `POST {dql_url}?token=…` with `{"query", "size", "from"?,
"format"?, "filter"?, "exportspec"?, …extra}`. The token stays in the query string;
the query text no longer appears in URLs, and the URL length limit is gone. `extra` is
typed `Mapping[str, Any]` and its values go into the JSON body verbatim.
- A failed crawl job raises `JobError` (carrying the job name) instead of a synthesized
`APIError(500, ...)`. `JobError` subclasses `APIError`, so `except APIError` still
catches it.
- Internal: shared sync/async transport helpers in `_transport.py` replace the
per-handler `_http.get(...)` + raise pattern.

### Fixed

- `ask()`/`ask_json()` (sync and async): a non-2xx response while streaming raises
`AuthError`/`RateLimitError`/`APIError` as documented instead of
`httpx.ResponseNotRead`.
- `dql(format="csv")` and `dql(format="jsonl")` without `raw=True` crashed in
`response.json()`. `csv` now returns `str`, `xls`/`xlsx` return `bytes`, and `jsonl`
returns an iterator of entities (`AsyncIterator` on the async client).
- `crawl()` no longer reports `JOB_CREATED` for a create that failed with HTTP 200 +
`errorCode`.
- `APIError.message` falls back to the body's `error` field (the `{errorCode, error}`
shape), matching the TypeScript SDK.
- `Retry-After: inf` no longer crashes `time.sleep` with `OverflowError`; non-finite
values fall back to backoff.

### Known limitations

- `extra` can still override the pre-existing DQL keys (`size`, `from`, `format`,
`filter`, `exportspec`); only the new named keys are collision-checked.
- `crawl()` has no `extra`; recurring-crawl admin (`repeat`, `maxRounds`, ...) is not
exposed. `Diffbot.crawl(**kwargs)` does not type-check the new create options.
- `nlp()` returns one `NlpResponse` `TypedDict` (all fields optional), not a per-`fields`
type. `custom_summary` reads only the Python key `max_number_of_sentences`.
- `crawl_get_job` still returns `{}` for a missing job (the TypeScript SDK returns
`undefined`).
- No generic success-path response headers.

### Migrating DQL: mocks and proxies

Test mocks keyed on `GET` and `?query=` stop matching. Before/after:

```python
# before (3.1.x)
def handler(request: httpx.Request) -> httpx.Response:
assert request.method == "GET"
assert request.url.params["query"] == "type:Organization"
return httpx.Response(200, json={"data": []})

# after (3.4.0)
def handler(request: httpx.Request) -> httpx.Response:
assert request.method == "POST"
assert json.loads(request.content)["query"] == "type:Organization"
return httpx.Response(200, json={"data": []})
```

GET-only proxies, caches, or a custom `dql_url` mirror may reject or not cache
POSTs. If you need the old wire format, use the low-level call (no new API):

```python
db.request("GET", db.dql_url, params={"token": db.token, "query": q, "size": 10})
```

or pin `diffbot<3.4` (3.1.x).
Loading
Loading