Repository navigation
Call the Diffbot APIs through the diffbot Python SDK (3.4.1); release 0.3.0 - #7
Merged
Merged
Conversation
Replace the aiohttp helper with a per-call diffbot.DiffbotAsync client (diffbot~=3.4.0). Extract, web search, entities, DQL, the ontology and the crawl job actions now go through the SDK, which brings DQL as a POST with a JSON body (no URL length limit on long queries), retries on 429/5xx and connection errors, and typed errors mapped to the same ToolError messages. Crawl start still builds its own create params, sent through the SDK transport: the SDK omits obeyRobots when it is false, and the crawler's default is to obey robots.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 0.3.0
diffbot 3.4.1 sends obeyRobots=0 for an explicit obey_robots=False, so the
hand-built create request is no longer needed. Start now calls db.crawl()
on a client with retries off (creating a job is not idempotent) and
returns {job_name, created}. Versions stamped with scripts/sync_version.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shixish
force-pushed
the
refactor/use-python-sdk
branch
from
November 18, 2026 22:55
aac14a4 to
1fb5305
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the backlog item "Move diffbot-mcp onto the Python SDK". Approved by Andrew (Nov 18) to merge and release as 0.3.0.
What changes
The aiohttp helper in
diffbot_mcp/api.pyis replaced bydiffbot(), an async context manager that yields adiffbot.DiffbotAsyncclient for one tool call. The client is per call because the token is per request on the HTTP transport. The dependencyaiohttp~=3.14.5becomesdiffbotdiffbot>=3.4.1(CI was first verified against the pinneddiffbot==3.4.1rc1).extract/v3/{api}by hand, checkerrorCodedb.extract(url, api=..., fmt=...);ExtractionErrorkeeps the same hint messagesearch_webrerank=nonefor fulldb.web_search(...). The SDK setsrerank=noneforsnippet_length="full".build_search_paramsis nowdefault_snippet_lengthresolve_entitiesnl.diffbot.com/v1/by handdb.entities(text, lang=...)dql,dql_probe, the typed search tools?query=db.dql(...), sent as POST with a JSON body, so a long query can't hit a URL length limit. CSV viaformat="csv", exportspec=...dql_ontology/kg/ontologydb.dql_fetch_ontology_text(), still cached for a day. The endpoint is public, so a missing token still doesn't block itcrawlstatus / list / delete / urls / results/v3/crawland/v3/crawl/databy handcrawl_get_job,crawl_list_jobs,crawl_delete_job(not retried),crawl_downloadcrawlstart/v3/crawldb.crawl(url, ..., watch=False)on a client withretry=False(creating a job is not idempotent). Same wire params as before (obeyRobots=0/useProxies=0are sent explicitly, which needs diffbot ≥ 3.4.1). Raises on a 200 that carrieserrorCode(e.g. a job name over 30 characters). Returns{"job_name", "created": true}instead of the raw create bodyErrors: SDK exceptions are mapped onto the same
ToolErrormessages as before, by type (AuthError→ "Invalid or unauthorized Diffbot token: …",RateLimitError→ "… retry after Ns",APIError/JobError→ "Diffbot API error N: …", httpx timeout / transport errors).Behavior differences to review
RetryPolicy). The old helper never retried. Crawl create and delete are not retried.db.crawl(). The earlier hand-built request worked around the SDK droppingobeyRobotswhenobey_robots=False; that is fixed in diffbot 3.4.1 (fix(crawl): send obeyRobots=0 / useProxies=0 when explicitly False diffbot-python#13), so the workaround is gone.crawlstartnow returns{"job_name", "created": true}instead of the raw create response body (the SDK'scrawl()only yields the job name).crawldeletenow returns{"job_name", "deleted": true}instead of the raw API body.resultsfor an unknown job now reports the SDK'sJobErrormessage.CI
This PR adds
.github/workflows/tests.yml, which runs the unit tests (tests/test_auth.py,tests/test_builder.py; no network or token) on 3.10 and 3.12. Before this PR, no workflow ran the tests on a PR.test_auth.pynow sends extract through the real SDK client over anhttpx.MockTransport, and checks that the right per-request token reaches the Diffbot API.Not run here:
tests/smoke.py(live APIs) andtests/eval_routing.py. Run the smoke test with a token before releasing.Version bumped to 0.3.0 with
scripts/sync_version.py(pyproject.toml, server.json, manifest.json, plugin.json, gemini-extension.json,diffbot_mcp/__init__.py). Rebased onto main after #8.🤖 Generated with Claude Code