Skip to content

Point diffbot at 3.4.0rc0 for CI verification - #1

Merged
shixish merged 15 commits into
mainfrom
feat/sdk-3.x-wiring
Nov 18, 2026
Merged

shixish merged 15 commits into
mainfrom
feat/sdk-3.x-wiring

Conversation

@shixish

@shixish shixish commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

RC verification for the unison release.

This branch pins diffbot==3.4.0rc0 in diffbot/requirements.txt and diffbot/pyproject.toml so GitHub CI can install the published Python SDK release candidate from PyPI and run the suite.

Do not merge until the stable 3.4.0 SDK is published and the pin is switched to ==3.4.0 for the unison / marketplace publish. The langgenius/dify-plugins marketplace PR is still not part of this work and still needs a separate approval.

shixish and others added 15 commits November 17, 2026 12:58
…I skip

Production pins stay on dify_plugin==0.11.0 with TODOs for diffbot 3.2.0/3.3.0.
requirements-dev.txt installs the local SDK checkout editable. CI cannot see it,
so tests/conftest.py skips SDK-dependent modules when diffbot is missing.
Drop requests, which the plugin no longer uses.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
make_client() validates token/timeout and returns a Diffbot client behind a
GuardedTransport that blocks private hosts. SDK errors map to redacted plugin
errors. verify_token, extract options and crawl_download feature-detect the SDK
3.3.0 methods and fall back to client.request(). Enhance stays a plugin-side
wrapper. Extract normalizes bare domains and gains caller-set options; web
search drops the implicit short snippet; KG tools gain offset and size up to 100.
Tests now run the real SDK over httpx.MockTransport.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d fields

crawl_start never watches and returns the job name. crawl_jobs lists, shows
status, fetches results via crawl_download (or the /v3/crawl/data fallback), and
deletes (labelled destructive, never retried). Provider gains request_timeout and
block_private_hosts. test_sdk_wiring runs against the SDK with and without the
3.3.0 methods.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The crawler answers a name over 30 characters with HTTP 200 + errorCode 505
and creates nothing (diffbot 3.2.0 does not surface that), so crawl_start
now rejects such names up front. crawl_jobs results asked for num=limit,
so 'truncated' could never be true; request limit+1 and default to the
yaml's 100 rows. Fix KG search llm_descriptions that said 1000 instead of 100.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rrorCode

- JobError carries plain text, so APIError.message was empty and users saw
  'Diffbot API error 500: request failed'; show the (redacted) reason.
- The crawl_download fallback returned an HTTP 200 + errorCode body (e.g. an
  unknown job) as a result row; raise JobError like diffbot 3.3.0 does.
- A non-JSON body escaped mapped_errors as a raw JSONDecodeError; map it.
- The Enhance wrapper ignored HTTP 200 + errorCode (the old client checked it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an check

Only CrawlJob and ExtractResponse are read by the plugin; the DqlResponse,
WebSearchResponse, and NlpResponse fallbacks were dead code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thon -m pytest

Without the SDK installed, 'import diffbot' resolves to the plugin's own
diffbot/ directory as a namespace package when the repo root is on sys.path,
so the SDK-dependent modules were not skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
httpx stops reading proxy environment variables once a transport is passed
explicitly, so wrapping httpx.HTTPTransport() in GuardedTransport silently
dropped the proxy support the old requests client had. Build per-scheme
proxied transports from the environment and respect NO_PROXY. Test mocks
accept transport kwargs so the suite also passes with a proxy set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the tool YAML/implementation, its provider entry, the plugin-side
enhance() transport wrapper and trim_enhance() shaper, the related tests,
and every mention in README, PRIVACY, and SUBMISSION docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Describe the eight current tools (enhance_entity removed in 7323b7b),
crawl_start/crawl_jobs behavior (background jobs, 30-character job
names, destructive delete, account-wide job scope), the diffbot SDK +
httpx transport replacing requests, the three credential fields, the
four network domains, and a crawl-aware high-risk narrative.

Drop the pre-SDK validator numbers. The package cannot be revalidated
until diffbot 3.4.0 is on PyPI, so those results are marked
TODO(sdk-release): rerun marketplace validator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drops the field, _as_bool and the toggle plumbing in GuardedTransport/make_client,
and the self-hosted gateway wording in docs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The plugin version stays 0.1.0. The editable workspace path is gone.
Marketplace submission stays a separate pull request.
The dependencies section still said the pin was not applied.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shixish
shixish marked this pull request as ready for review November 18, 2026 21:59
@shixish
shixish merged commit 73a275d into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant