Repository navigation
Release diffbot 3.4.0 - #10
Merged
Merged
Conversation
…ility set, retry policy, and error hierarchy Mirrors the TypeScript SDK's Phase 0 transport work (same branch) in Python idiom: - Shared sync/async request-shaping and response-handling helpers in a new _transport.py, replacing the duplicated Diffbot/DiffbotAsync._raise_for_status methods and the repeated client._http.get(...) + raise pattern across every handler module with a single client.request(method, url, ...) used by both. - Per-host base URLs (api_base/kg_base/llm_base/nlp_base) with the existing per-endpoint *_url constructor options kept as winning aliases (additive). No standalone "endpoint registry" data structure — Phase 0 adds zero new endpoints, so the only thing it would serve today is base-URL resolution (done directly in client.py) and a retryable flag (done ad hoc), matching the TypeScript SDK's same call on this branch (see its PHASE0_LEFTOVERS.md). - Repeated query parameters (declared via a Sequence-typed QueryParams; httpx already encodes list-valued params as repeated keys). - DELETE + arbitrary method/body/Content-Type, form-encoded (data=) and raw (content=) request bodies, per-call timeout override, and a follow_redirects override, all threaded through client.request(). - A default-on retry policy (RetryPolicy dataclass) for 429/5xx/connection errors with backoff, jitter, and Retry-After support, capped at 3 attempts; disable client-wide with retry=False or per call with retryable=False. crawl_delete_job/_async now pass retryable=False since it's a GET-shaped mutation that must not be retried blindly. - Response headers surfaced on APIError (and subclasses) via a new headers field, populated from the error response. - NotFoundError (404) and JobError, both subclassing APIError so existing `except APIError` call sites are unaffected. crawl's job-failure path now raises JobError instead of a synthesized APIError(500, ...). - fix(ask): read the response body before raising on a non-2xx streaming response (sync and async). Previously client._raise_for_status accessed response.text immediately after opening the stream, which raises httpx.ResponseNotRead instead of AuthError/RateLimitError once the connection is genuinely still streaming (tests use a real byte-stream mock transport, since httpx.MockTransport's content= pre-fills the body and masks this). ask()/ask_json() now go through client.request(..., stream=True), which also means they pick up the retry policy like every other handler. - web_search.py's sync/async bodies, previously 100% duplicated, now share a single _build_web_search_request() helper. Also adds additive TypedDict/Literal type aliases (types.py) for existing surfaces only — ChatMessage, ResponseFormat, ExportFormat, SnippetLength — used in ask()/dql()/web_search() signatures. No Pydantic, no runtime behavior change; Python floor stays >=3.9 (stdlib typing only). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Client configuration table and adds examples for api_base/kg_base/ llm_base/nlp_base (pointing the whole SDK at a staging host with one option) and retry/RetryPolicy, matching the TypeScript SDK's README for the same Phase 0 work. Adds the matching tests per AGENTS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… changelog entry PHASE0_LEFTOVERS.md: no standalone endpoint-registry data structure (same call as the TypeScript SDK on this branch — Phase 0 adds zero new endpoints, so a registry has no second consumer yet), response headers surfaced on the error path only (success-path headers deferred to Phase 6's dql_report, which doesn't exist yet to shape it against), confirmation that Python's dql(raw=True) never had the TypeScript SDK's binary-corruption bug, the ontology async/text asymmetry being Phase 1 scope rather than Phase 0, and the scope of the TypedDict/Literal additions (existing request parameters, not response shapes — this repo has no type-checking CI gate to shape response overlays against). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dql()/dql_parallel() and async twins now POST to the DQL endpoint with a JSON body; the token stays in the query string. Public contract unchanged. extra widens to Mapping[str, Any]. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Separate workflow, never on pull requests, concurrency of 1. Adds a cheap live DQL POST/entities suite asserting on shape only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on note Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Several Diffbot APIs answer with {errorCode, error} rather than {message}.
Mirror the TypeScript SDK's tryParseErrorBody so APIError.message (and str(err))
carry that text, and ignore non-string message/requestId values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocstring Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erify_token, crawl_download, extract options - types.py: stdlib TypedDict(total=False) response shapes (DqlResponse, WebSearchResponse, EntitiesResponse, ExtractResponse + per-API, CrawlJob, AccountResponse, AskChunk) and per-API extract option/field Literals. - dql: @Overloads by format/raw; csv -> str, jsonl -> iterator of entities (previously both crashed in response.json() without raw=True). - account(days=None) / verify_token() (GET /v4/account); only AuthError -> False. - crawl_download(name, format, type, num); the crawl watcher reuses it. - extract: keyword-only timeout/mode/fields/etc; transport timeout raised to at least timeout + 10s; extra for proxy/proxyAuth. - pyright dev dependency + config (basic on src/, strict on types.py), CLI annotation fixes, tests/typing assert_type checks. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, crawl_download Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, account/verify_token Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… type AskChunk in the SSE parser Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l create-error body
- crawl_download(type="urls") returned a JSON decode error: that report is always
CSV, whatever format says. Return the text (matching the TypeScript SDK) and add a
type="urls" -> str overload.
- crawl_download raised nothing for a non-list JSON body such as
{errorCode, error} for an unknown job and handed the dict back as records.
Raise JobError carrying the job name, as TypeScript does.
- crawl create with HTTP 200 + errorCode now raises APIError with the real status,
body and headers instead of a synthesized body; message comes from the error
field via the APIError fallback.
- Document the 30-character job name limit (README, crawl docstring) and add the
create-error fix to the 3.3.0 changelog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xportfile, jsonmode, non_canonical_facts, no_dedup_articles, cluster Keyword-only options on dql()/dql_async() (and so dql_parallel), sent in the POST body. Explicit option plus the same wire key in extra raises ValidationError. report/view stay reachable via extra only. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
url_crawl_regex, url_process_regex, page_process_pattern, max_to_crawl_per_subdomain, max_to_process_per_subdomain, use_canonical, notify_webhook, notify_email on crawl() (query params on the existing GET). custom_headers accepts a str or a sequence, sent as repeated params. Recurring-crawl admin stays out. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nlp(text, *, fields, lang, format, document_type, custom_summary) plus async twin, single document. Adds NlpField/NlpFormat/NlpCustomSummary/NlpResponse types, README snippet and its test. entities() keeps hard-coding fields=entities,sentiment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…arity text may be a str or 1-5 strings (repeated text= params on the existing GET); 0 or >5 raises ValidationError. snippet_length is case-normalized and validated like the TS SDK; invalid values now raise ValidationError. Docstrings label snippet_length/rerank as undocumented. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Closes the Python/TypeScript ontology asymmetry (PHASE0_LEFTOVERS §4). Raw JSON text fetch on both clients; async refresh on DiffbotAsync (dql_refresh_ontology) and in diffbot.kg (dql_refresh_ontology_async). dql_fetch_ontology reuses the text fetch. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd async Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 429/5xx Retry-After was honored verbatim, so `Retry-After: 3600` made one call sleep an hour; max_delay capped only the exponential backoff. Now a Retry-After (seconds or HTTP-date) above RetryPolicy.max_delay means no sleep and no retry: the response is raised at once as RateLimitError with retry_after populated (or APIError with the header for a 5xx), in the sync, async and streamed paths. Values at or under the cap are honored as before. Non-finite seconds (inf/nan) fall back to backoff instead of crashing time.sleep. Matches the TypeScript SDK. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump pyproject.toml and the uv.lock entry for the project from 3.1.0 to 3.4.0. __version__, the db --version string, and the User-Agent all read installed metadata, so there is no second constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.2.0 and 3.3.0 were never published, so the changelog now has a single 3.4.0 entry (Keep a Changelog) covering everything since 3.1.x, with the breaking changes listed first: DQL over POST, crawl create HTTP 200 + errorCode now raises, Retry-After over max_delay is not retried, retries are on by default, snippet_length is validated, and dql extra collisions raise. PHASE0/S1/S2_LEFTOVERS.md recorded only closed items, deliberate narrowings, and live checks that have now run. The narrowings are under "Known limitations" in the changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.1.1 is on PyPI but feat/sdk-s2 forked from 3.1.0, so releasing 3.4.0 without this would undo it: db -o now creates missing parent folders, the verify-release targets refresh uv's index cache, and the macOS release binary builds on macos-15. Conflicts were only the version line in pyproject.toml and uv.lock; 3.4.0 is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add .github/workflows/publish-pypi.yml: verify (tag == pyproject version == diffbot.__version__, unit tests, pyright), build (uv build sdist + wheel, twine check --strict, artifact), and publish (tags only, environment "pypi", id-token: write, pypa/gh-action-pypi-publish@release/v1; no API token). workflow_dispatch is a dry run that never uploads. Concurrency never cancels an in-flight publish. release.yml (binaries) is unchanged. Document the release path (merge to main, push tag vX.Y.Z) in the README and mark `make release` as a manual fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One Release workflow on v* tags: verify, build sdist/wheel and db binaries, publish to PyPI via trusted publishing (after binaries build), then the GitHub Release. Concurrency never cancels an in-flight publish. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
diffbot 3.4.0
The first release since 3.1.1. It adds a shared transport with retries, response
TypedDicts with a pyright gate, DQL over POST,nlp(), and named options on the endpoints the SDK already covers. It includes the 3.2.0 and 3.3.0 work, which was never published on its own.The self-contained
dbbinaries for Linux (x86_64 and aarch64) and macOS (Apple Silicon) are attached to this release.⚠ Breaking changes
POSTwith a JSON body.dql(),dql_parallel()and their async versions keep the same signature and return values. The token stays in the query string, and the query text moves into the body, which removes the URL length limit. Test mocks, proxies and mirrors that match onGET+?query=need updating. See the CHANGELOG migration section. A GET escape hatch is still available:db.request("GET", db.dql_url, params=...).crawl()create now raises. If the crawler answers HTTP 200 witherrorCode/error(for example errorCode 505 for a job name over 30 characters),crawl()raisesAPIError. It used to yieldJOB_CREATEDfor a job that was never created.429,5xxand connection errors get up to 3 attempts with backoff and jitter.crawl_delete_jobis never retried. Passretry=Falseto opt out.Retry-Aftervalues are no longer waited out. If the header asks for longer thanRetryPolicy.max_delay(default 5 s), the call raises at once withRateLimitError(retry_afterholds the raw header) or, for a 5xx, withAPIError. This applies to sync, async and streamed calls. Raisemax_delayif you want to keep waiting.web_search(snippet_length=...)is validated. Values other thanshort,medium,longorfullraiseValidationError.dql()raisesValidationErrorifextrarepeats one of the new named options.dql,extract,web_search,entitiesand thecrawl_*getters returnTypedDicts. Runtime values are unchanged. Usetyping.cast(Dict[str, Any], ...)if a type checker objects.New
account(days=...)andverify_token(), sync and async.crawl_download(name, format=..., type=..., num=...), sync and async. A JSON body for an unknown job raisesJobError.extract()keyword options fortimeout,mode, per-APIfields,paging,max_pages,use_proxyand more, plusextra.type(including"crawl"together withcol),filter_exclude,exportseparator,exportfile,jsonmode,non_canonical_facts,no_dedup_articles,cluster.url_crawl_regex,url_process_regex,page_process_pattern, per-subdomain limits,use_canonical,notify_webhook,notify_email.custom_headersalso accepts a list.nlp(text, fields=[...], ...), sync and async.web_search()accepts 1–5textvalues.dql_fetch_ontology_text(), anddql_refresh_ontologyonDiffbotAsync.api_base,kg_base,llm_base,nlp_base),client.request(...)for any method or body with a per-calltimeout,NotFoundError/JobError(both subclasses ofAPIError), andAPIError.headers.TypedDicts exported fromdiffbot. They use the stdlib only and work on Python 3.9.Fixes
dql(format="csv")anddql(format="jsonl")no longer crash: they returnstrand an iterator of entities.ask()/ask_json()raisesAuthError/RateLimitErrorinstead ofhttpx.ResponseNotRead.APIError.messagefalls back to theerrorfield of{errorCode, error}bodies.Retry-After: infno longer crashestime.sleep.The full list is in CHANGELOG.md.