All notable changes to the diffbot Python package are documented here. The format
follows Keep a Changelog, and the project uses
Semantic Versioning.
Same code as 3.4.1rc1, which was verified against diffbot-mcp CI.
crawl(obey_robots=False)andcrawl(use_proxies=False)were silently ignored. An explicitFalsewas dropped instead of sent, so the API default applied (forobeyRobotsthat means robots.txt was still obeyed). Both are now tri-state likeuse_canonical:None(the new default) omits the param,True/Falsesend1/0. Sync and async.- CLI
db crawl:--obey-robots/--use-proxiesgain--no-obey-robots/--no-use-proxies, and--restrict-domaingains--no-restrict-domain(it could not be turned off before). Omitting the robots/proxies flags still leaves the API default.
First release since 3.1.1 (and includes everything in it). It rolls up three development lines that were never
published on their own: the shared transport, sync/async de-duplication and DQL over
POST (planned as 3.2.0), response types plus account/verify_token/crawl_download/
extract options (3.3.0), and named options on existing endpoints plus nlp() (3.4.0).
No new Diffbot API areas are added.
- DQL is sent as
POSTwith a JSON body.dql(),dql_parallel()and their async twins keep the same signature and results, but mocks, proxies, or mirrors keyed onGET+?query=stop matching. See Migrating DQL below. crawl()raisesAPIErrorwhen the create call answers HTTP 200 witherrorCode/error(for example errorCode 505, job name over 30 characters). It used to yieldJOB_CREATEDfor a job that was never created. The error keeps the real status (200), body, and headers.- A
Retry-AfteraboveRetryPolicy.max_delay(default 5.0 s) is no longer waited out or retried. The call raises at once:RateLimitErrorwithretry_after= raw header, orAPIErrorwithheaders["retry-after"]for a 5xx (sync, async, and streamed). Raisemax_delayto keep waiting. - Requests are now retried by default (
429,5xx, connection errors; 3 attempts with backoff and jitter).crawl_delete_jobis never retried. Passretry=Falsefor the old single-attempt behavior. web_search(snippet_length=...)is validated: anything other thanshort,medium,long,full(case-insensitive) raisesValidationErrorinstead of being sent upstream.dql()raisesValidationErrorwhenextrarepeats one of the new named options (type,col,filter_exclude, ...).- Type-level:
dql,extract,web_search,entities,crawl_list_jobs,crawl_get_jobreturnTypedDicts instead ofDict[str, Any], and several methods gained@overloads. Runtime values are unchanged; a type checker may flag undeclared keys (escape hatch:typing.cast(Dict[str, Any], resp)).
- Transport: per-host base URLs (
api_base,kg_base,llm_base,nlp_baseonDiffbotandDiffbotAsync; the older*_urloptions still win when both are set); documented repeated query params;client.request(method, url, ...)withDELETE, form (data=) and raw (content=) bodies, a per-calltimeout, and afollow_redirectsoverride; theRetryPolicydataclass andretryoption with per-callretryable=False. - Errors:
NotFoundError(404) andJobError, both subclasses ofAPIError;APIError.headersfrom the error response. - Types: request aliases (
ChatMessage,ResponseFormat,ExportFormat,SnippetLength) and responseTypedDicts exported fromdiffbot(DqlResponse/DqlEntity,WebSearchResponse/WebSearchResult,EntitiesResponse,ExtractResponseand per-API variants,CrawlJob,CrawlDataRecord,AccountResponse,AskChunk, ...). stdlib only, Python 3.9-compatible, no Pydantic. account(days=None)andverify_token()(sync and async) overGET /v4/account.verify_token()returnsFalseonly on an auth failure. The account response carriestokenandchildTokens; don't log it whole.crawl_download(name, *, format="json", type=None, num=None)(sync and async) overGET /v3/crawl/data.format="csv"ortype="urls"returnsstr, otherwise a list of records; a non-list JSON body (e.g. unknown job) raisesJobError. The crawl watcher uses it. For tools that can't block:crawl(..., watch=False)->crawl_get_job->crawl_download.extract()keyword-only options:timeout(server-side ms, default 30000),mode,fields(per-APILiteralnames),fallback,discussion,paging,max_pages,natural_language,summary_num_sentences,max_tags,tag_confidence,category_confidence,use_proxy, andextra(proxy/proxyAuth; collisions with a named option ortoken/urlraiseValidationError). Settingtimeoutraises the per-call transport timeout to at leasttimeout + 10s.dql()/dql_async()(anddql_parallel) keyword-only optionstype("query","text","queryTextFallback","crawl"),col,filter_exclude,exportseparator,exportfile,jsonmode,non_canonical_facts,no_dedup_articles,cluster, sent in the POST body.reportandviewstay reachable viaextra. Types:DqlType,DqlJsonMode,DqlCluster,DqlOptions.crawl()create optionsurl_crawl_regex,url_process_regex,page_process_pattern,max_to_crawl_per_subdomain,max_to_process_per_subdomain,use_canonical,notify_webhook,notify_email;custom_headersaccepts a string or a sequence (repeated params). The 30-character job-name limit is documented.nlp(text, *, fields, lang, format, document_type, custom_summary)(sync and async) on the existing NL endpoint, single document.entities()is unchanged. Types:NlpField,NlpFormat,NlpCustomSummary,NlpResponse.web_search(text=...)accepts a string or 1-5 strings (repeatedtext=); out-of-range lengths raiseValidationErrorbefore any request.dql_fetch_ontology_text()(sync and async);dql_refresh_ontologyonDiffbotAsync(module functiondiffbot.kg.dql_refresh_ontology_async).- Dev:
pyrightgate (basic onsrc/, strict ontypes.py, plustests/typing) in CI; live test workflow (push tomain, manual dispatch, twice-monthly cron; never on PRs).
dql()request shape:POST {dql_url}?token=…with{"query", "size", "from"?, "format"?, "filter"?, "exportspec"?, …extra}. The token stays in the query string; the query text no longer appears in URLs, and the URL length limit is gone.extrais typedMapping[str, Any]and its values go into the JSON body verbatim.- A failed crawl job raises
JobError(carrying the job name) instead of a synthesizedAPIError(500, ...).JobErrorsubclassesAPIError, soexcept APIErrorstill catches it. - Internal: shared sync/async transport helpers in
_transport.pyreplace the per-handler_http.get(...)+ raise pattern.
ask()/ask_json()(sync and async): a non-2xx response while streaming raisesAuthError/RateLimitError/APIErroras documented instead ofhttpx.ResponseNotRead.dql(format="csv")anddql(format="jsonl")withoutraw=Truecrashed inresponse.json().csvnow returnsstr,xls/xlsxreturnbytes, andjsonlreturns an iterator of entities (AsyncIteratoron the async client).crawl()no longer reportsJOB_CREATEDfor a create that failed with HTTP 200 +errorCode.APIError.messagefalls back to the body'serrorfield (the{errorCode, error}shape), matching the TypeScript SDK.Retry-After: infno longer crashestime.sleepwithOverflowError; non-finite values fall back to backoff.
extracan still override the pre-existing DQL keys (size,from,format,filter,exportspec); only the new named keys are collision-checked.crawl()has noextra; recurring-crawl admin (repeat,maxRounds, ...) is not exposed.Diffbot.crawl(**kwargs)does not type-check the new create options.nlp()returns oneNlpResponseTypedDict(all fields optional), not a per-fieldstype.custom_summaryreads only the Python keymax_number_of_sentences.crawl_get_jobstill returns{}for a missing job (the TypeScript SDK returnsundefined).- No generic success-path response headers.
Test mocks keyed on GET and ?query= stop matching. Before/after:
# before (3.1.x)
def handler(request: httpx.Request) -> httpx.Response:
assert request.method == "GET"
assert request.url.params["query"] == "type:Organization"
return httpx.Response(200, json={"data": []})
# after (3.4.0)
def handler(request: httpx.Request) -> httpx.Response:
assert request.method == "POST"
assert json.loads(request.content)["query"] == "type:Organization"
return httpx.Response(200, json={"data": []})GET-only proxies, caches, or a custom dql_url mirror may reject or not cache
POSTs. If you need the old wire format, use the low-level call (no new API):
db.request("GET", db.dql_url, params={"token": db.token, "query": q, "size": 10})or pin diffbot<3.4 (3.1.x).