Skip to content

Call the Diffbot APIs through the diffbot Python SDK (3.4.1); release 0.3.0 - #7

Merged
shixish merged 4 commits into
mainfrom
refactor/use-python-sdk
Nov 18, 2026
Merged

shixish merged 4 commits into
mainfrom
refactor/use-python-sdk

Conversation

@shixish

@shixish shixish commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes the backlog item "Move diffbot-mcp onto the Python SDK". Approved by Andrew (Nov 18) to merge and release as 0.3.0.

What changes

The aiohttp helper in diffbot_mcp/api.py is replaced by diffbot(), an async context manager that yields a diffbot.DiffbotAsync client for one tool call. The client is per call because the token is per request on the HTTP transport. The dependency aiohttp~=3.14.5 becomes diffbot diffbot>=3.4.1 (CI was first verified against the pinned diffbot==3.4.1rc1).

Tool / path Before After
extract GET /v3/{api} by hand, check errorCode db.extract(url, api=..., fmt=...); ExtractionError keeps the same hint message
search_web hand-built params, rerank=none for full db.web_search(...). The SDK sets rerank=none for snippet_length="full". build_search_params is now default_snippet_length
resolve_entities POST nl.diffbot.com/v1/ by hand db.entities(text, lang=...)
dql, dql_probe, the typed search tools GET with ?query= db.dql(...), sent as POST with a JSON body, so a long query can't hit a URL length limit. CSV via format="csv", exportspec=...
dql_ontology GET /kg/ontology db.dql_fetch_ontology_text(), still cached for a day. The endpoint is public, so a missing token still doesn't block it
crawl status / list / delete / urls / results GET /v3/crawl and /v3/crawl/data by hand crawl_get_job, crawl_list_jobs, crawl_delete_job (not retried), crawl_download
crawl start GET /v3/crawl db.crawl(url, ..., watch=False) on a client with retry=False (creating a job is not idempotent). Same wire params as before (obeyRobots=0/useProxies=0 are sent explicitly, which needs diffbot ≥ 3.4.1). Raises on a 200 that carries errorCode (e.g. a job name over 30 characters). Returns {"job_name", "created": true} instead of the raw create body

Errors: SDK exceptions are mapped onto the same ToolError messages as before, by type (AuthError → "Invalid or unauthorized Diffbot token: …", RateLimitError → "… retry after Ns", APIError/JobError → "Diffbot API error N: …", httpx timeout / transport errors).

Behavior differences to review

  • Retries. The SDK retries 429, 5xx, and connection errors with backoff (its default RetryPolicy). The old helper never retried. Crawl create and delete are not retried.
  • Crawl start uses db.crawl(). The earlier hand-built request worked around the SDK dropping obeyRobots when obey_robots=False; that is fixed in diffbot 3.4.1 (fix(crawl): send obeyRobots=0 / useProxies=0 when explicitly False diffbot-python#13), so the workaround is gone.
  • crawl start now returns {"job_name", "created": true} instead of the raw create response body (the SDK's crawl() only yields the job name).
  • crawl delete now returns {"job_name", "deleted": true} instead of the raw API body. results for an unknown job now reports the SDK's JobError message.
  • The transport is httpx (via the SDK) instead of aiohttp. The total timeout stays at 120s.

CI

This PR adds .github/workflows/tests.yml, which runs the unit tests (tests/test_auth.py, tests/test_builder.py; no network or token) on 3.10 and 3.12. Before this PR, no workflow ran the tests on a PR. test_auth.py now sends extract through the real SDK client over an httpx.MockTransport, and checks that the right per-request token reaches the Diffbot API.

Not run here: tests/smoke.py (live APIs) and tests/eval_routing.py. Run the smoke test with a token before releasing.

Version bumped to 0.3.0 with scripts/sync_version.py (pyproject.toml, server.json, manifest.json, plugin.json, gemini-extension.json, diffbot_mcp/__init__.py). Rebased onto main after #8.

🤖 Generated with Claude Code

shixish and others added 3 commits November 18, 2026 12:51
Replace the aiohttp helper with a per-call diffbot.DiffbotAsync client
(diffbot~=3.4.0). Extract, web search, entities, DQL, the ontology and the
crawl job actions now go through the SDK, which brings DQL as a POST with a
JSON body (no URL length limit on long queries), retries on 429/5xx and
connection errors, and typed errors mapped to the same ToolError messages.

Crawl start still builds its own create params, sent through the SDK
transport: the SDK omits obeyRobots when it is false, and the crawler's
default is to obey robots.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… 0.3.0

diffbot 3.4.1 sends obeyRobots=0 for an explicit obey_robots=False, so the
hand-built create request is no longer needed. Start now calls db.crawl()
on a client with retries off (creating a job is not idempotent) and
returns {job_name, created}. Versions stamped with scripts/sync_version.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shixish
shixish force-pushed the refactor/use-python-sdk branch from aac14a4 to 1fb5305 Compare November 18, 2026 22:55
@shixish shixish changed the title Call the Diffbot APIs through the diffbot Python SDK (3.4.0) Call the Diffbot APIs through the diffbot Python SDK (3.4.1); release 0.3.0 Nov 18, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shixish
shixish merged commit fed35d6 into main Nov 18, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant