From 4391ecdf27e0909456573a5d3e13c7eadee46402 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 12:58:48 -1000 Subject: [PATCH 01/15] Wire dependencies for the Diffbot SDK: dev path install, TODO pins, CI skip Production pins stay on dify_plugin==0.11.0 with TODOs for diffbot 3.2.0/3.3.0. requirements-dev.txt installs the local SDK checkout editable. CI cannot see it, so tests/conftest.py skips SDK-dependent modules when diffbot is missing. Drop requests, which the plugin no longer uses. Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/test.yml | 4 ++++ README.md | 23 ++++++++++++++++------- diffbot/README.md | 11 +++++++---- diffbot/pyproject.toml | 2 +- diffbot/requirements.txt | 6 +++++- requirements-dev.txt | 9 +++++++++ tests/conftest.py | 12 ++++++++++++ 7 files changed, 54 insertions(+), 13 deletions(-) create mode 100644 requirements-dev.txt create mode 100644 tests/conftest.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index ce2791a..8eaec56 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -20,4 +20,8 @@ jobs: cache: pip cache-dependency-path: diffbot/requirements.txt - run: pip install -r diffbot/requirements.txt pytest + # TODO(release): the plugin needs the `diffbot` SDK (3.2.0, then 3.3.0), which is not + # published yet and cannot be installed from a private path or unpushed ref here. + # Until then tests/conftest.py skips the SDK-dependent modules. After release, pin + # `diffbot` in diffbot/requirements.txt (installed by the line above) and delete conftest.py. - run: pytest -q diff --git a/README.md b/README.md index b4f8a4e..f4615da 100644 --- a/README.md +++ b/README.md @@ -14,26 +14,35 @@ The installable plugin is in [`diffbot/`](diffbot/). Its [README](diffbot/README | --- | --- | | Web search | `GET https://llm.diffbot.com/api/v1/web_search` (bearer token) | | Extract page | `GET https://api.diffbot.com/v3/{analyze\|article\|product\|discussion\|image\|video\|list\|event}` (`token` query parameter) | -| Search organizations | `GET https://kg.diffbot.com/kg/v3/dql` | +| Search organizations | `POST https://kg.diffbot.com/kg/v3/dql` | | Search people | same DQL endpoint, public professional profiles only | | Search news | same DQL endpoint, newest publication date first via `sortBy:date` | -| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` | +| Start crawl | `GET https://api.diffbot.com/v3/crawl` (create only; returns a job name) | +| Crawl jobs | list / status / results / **delete (destructive)** on `/v3/crawl` and `/v3/crawl/data` | +| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` (plugin-side wrapper; not in the SDK) | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Saving the credential checks `GET https://api.diffbot.com/v4/account?days=1`. Get a free token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). -Crawl, raw DQL, ontology lookup, places, and deals are not in v0.1. See SUBMISSION.md for why. +Raw DQL, ontology lookup, Ask, places, and deals are not in this version. See SUBMISSION.md for why. + +All HTTP goes through the [Diffbot Python SDK](https://github.com/diffbot/diffbot-python) (`diffbot.Diffbot`), behind a transport that refuses private, loopback, and cloud-metadata hosts. ## Run the tests -Python 3.12: +Python 3.12+: ```bash -pip install -r diffbot/requirements.txt pytest -pytest +python -m venv .venv && . .venv/bin/activate +pip install -r requirements-dev.txt # dify_plugin, pytest, and the local SDK checkout +pytest -q ``` -The tests mock HTTP and make no Diffbot calls. `diffbot/requirements.txt` pins `dify_plugin==0.11.0` and `requests==2.34.2`. +The plugin imports the `diffbot` SDK, which is not released at the needed version yet (3.2.0, then 3.3.0). `diffbot/requirements.txt` therefore pins only `dify_plugin==0.11.0` and carries a TODO for the `diffbot` pin. `requirements-dev.txt` installs the local, unpushed SDK checkout editable (`-e /workspace/diffbot-python`); edit that path if your checkout lives elsewhere. + +The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). + +GitHub CI (`.github/workflows/test.yml`) cannot see the local path. Until `diffbot` is released it does not install the SDK, and `tests/conftest.py` skips the SDK-dependent modules (the DQL and URL-policy tests still run). The TODOs in both files say what to delete after release. ## Package diff --git a/diffbot/README.md b/diffbot/README.md index f777a66..dd3dfca 100644 --- a/diffbot/README.md +++ b/diffbot/README.md @@ -12,19 +12,22 @@ Published by Diffbot. Source: https://github.com/diffbot/dify-diffbot. Support: | --- | --- | --- | | Web search | `GET https://llm.diffbot.com/api/v1/web_search` | Ranked pages with a score, URL, date, and passage. A query that is only `url:https://...` reads the indexed copy. | | Extract page | `GET https://api.diffbot.com/v3/analyze` or a typed extract path | Live extraction of one public page. `article` is the usual choice for news and blogs. | -| Search organizations | `GET https://kg.diffbot.com/kg/v3/dql` | Companies by name, homepage, category, headquarters, headcount, revenue, or public status. | +| Search organizations | `POST https://kg.diffbot.com/kg/v3/dql` | Companies by name, homepage, category, headquarters, headcount, revenue, or public status. | | Search people | same DQL endpoint | Public professional profiles by name, employer, title, or location. | | Search news | same DQL endpoint | Articles by mentioned entity, phrase, publisher, language, or date. | +| Start crawl | `GET https://api.diffbot.com/v3/crawl` | Start a background crawl of public seed URLs. Returns a job name, never results. | +| Crawl jobs | `/v3/crawl`, `/v3/crawl/data` | List jobs, check status, fetch results (up to 1000 rows, truncated), or delete a job. **Delete is destructive** and removes the job and its data. | | Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` | Best Knowledge Graph match for a person or organization from a name, URL, or employer. | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Names in a passage, linked to Knowledge Graph ids when Diffbot can match them. | -Crawl, raw DQL, ontology lookup, places, and deals are not in this version. Industry filters must already be exact category names such as `Software Companies`. The plugin does not resolve free text onto the taxonomy. +Raw DQL, ontology lookup, Ask, places, and deals are not in this version. Industry filters must already be exact category names such as `Software Companies`. The plugin does not resolve free text onto the taxonomy. ## Credentials 1. Create a token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). 2. In Dify, open Plugins, install this package, and choose Diffbot. 3. Paste the token into **Diffbot token**. +4. Optional: set **Request timeout** (5 to 300 seconds, default 30). **Block private hosts** stays on unless you run a self-hosted gateway. Dify stores the value as a secret. The plugin sends it only to the Diffbot hosts listed in `manifest.yaml`. Web Search uses `Authorization: Bearer`. The other calls use the `token` query parameter, which is how those APIs authenticate. Saving the credential calls `GET https://api.diffbot.com/v4/account` with `days=1` and does not show the account payload. @@ -32,9 +35,9 @@ The token needs access to Extract, Web Search, Knowledge Graph, Enhance, and Nat ## Use the tools -In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits are small on purpose: at most 10 search rows, 3 enhance matches, and 20,000 characters of text for entity resolution. +In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits: up to 50 web results, 100 Knowledge Graph rows per call (page with Offset), 3 enhance matches, and 20,000 characters of text for entity resolution. Extract has optional caller-set options (fields, timeout, mode, paging, natural language, and others). -Extract accepts only `http` and `https` URLs. Localhost, link-local, and private literal addresses are rejected before the request is sent. +Extract accepts `http` and `https` URLs; a bare domain such as `example.com` gets `https://`. Localhost, link-local, and private literal addresses are rejected before the request is sent. ## Install the package locally diff --git a/diffbot/pyproject.toml b/diffbot/pyproject.toml index e26a969..3c3ee20 100644 --- a/diffbot/pyproject.toml +++ b/diffbot/pyproject.toml @@ -6,5 +6,5 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "dify_plugin==0.11.0", - "requests==2.34.2", + # TODO(release): add "diffbot==3.2.0", later "diffbot==3.3.0" (not published yet). ] diff --git a/diffbot/requirements.txt b/diffbot/requirements.txt index c0ee9fe..f0a99a8 100644 --- a/diffbot/requirements.txt +++ b/diffbot/requirements.txt @@ -1,2 +1,6 @@ dify_plugin==0.11.0 -requests==2.34.2 +# TODO(release): the plugin imports the `diffbot` SDK. Pin it here and in +# pyproject.toml once released: diffbot==3.2.0 (transport, retries, DQL POST), then +# diffbot==3.3.0 (verify_token, crawl_download, extract options, response types). +# Never list both `diffbot` and `diffbot-python`. +# Until then, local/dev runs use requirements-dev.txt (editable local SDK checkout). diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..67674f8 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,9 @@ +# Dev/test-only requirements. Not used for the production package. +# +# Installs the local, unreleased Diffbot Python SDK checkout (editable) so the +# plugin can be exercised against it on this machine. GitHub CI cannot see this +# path and does not use this file; CI stays on mocks until `diffbot` is released. +# TODO: delete the -e line once diffbot>=3.3.0 is on PyPI and pinned in diffbot/requirements.txt. +-r diffbot/requirements.txt +pytest +-e /workspace/diffbot-python diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..3e15f71 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,12 @@ +"""Skip SDK-dependent test modules when the ``diffbot`` SDK is not installed. + +GitHub CI cannot reach the unreleased local SDK (see requirements-dev.txt), so there +only the SDK-independent policy tests run. TODO: once diffbot 3.2.0/3.3.0 is on PyPI, +pin it in diffbot/requirements.txt and delete this file so every test runs in CI. +""" + +try: + import diffbot # noqa: F401 +except ImportError: + collect_ignore = ["test_tools_http.py", "test_sdk_wiring.py"] + print("diffbot SDK not installed: skipping SDK-dependent tests (see tests/conftest.py)") From 049dd29973a6c8072957a10d640c9beab4ff9729 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 12:58:48 -1000 Subject: [PATCH 02/15] Rebuild tool HTTP on the Diffbot SDK client make_client() validates token/timeout and returns a Diffbot client behind a GuardedTransport that blocks private hosts. SDK errors map to redacted plugin errors. verify_token, extract options and crawl_download feature-detect the SDK 3.3.0 methods and fall back to client.request(). Enhance stays a plugin-side wrapper. Extract normalizes bare domains and gains caller-set options; web search drops the implicit short snippet; KG tools gain offset and size up to 100. Tests now run the real SDK over httpx.MockTransport. Co-Authored-By: Claude Sonnet 5.5 --- diffbot/provider/diffbot.py | 9 +- diffbot/tools/client.py | 221 +++-------------- diffbot/tools/common.py | 26 +- diffbot/tools/enhance_entity.py | 9 +- diffbot/tools/extract.py | 79 ++++++- diffbot/tools/extract.yaml | 187 ++++++++++++++- diffbot/tools/resolve_entities.py | 6 +- diffbot/tools/sdk.py | 302 ++++++++++++++++++++++++ diffbot/tools/sdk_types.py | 63 +++++ diffbot/tools/search_news.py | 9 +- diffbot/tools/search_news.yaml | 18 +- diffbot/tools/search_organizations.py | 9 +- diffbot/tools/search_organizations.yaml | 18 +- diffbot/tools/search_people.py | 9 +- diffbot/tools/search_people.yaml | 18 +- diffbot/tools/search_web.py | 34 +-- diffbot/tools/search_web.yaml | 14 +- diffbot/tools/shape.py | 4 +- tests/test_tools_http.py | 68 +++--- 19 files changed, 802 insertions(+), 301 deletions(-) create mode 100644 diffbot/tools/sdk.py create mode 100644 diffbot/tools/sdk_types.py diff --git a/diffbot/provider/diffbot.py b/diffbot/provider/diffbot.py index 769c987..b4e6d98 100644 --- a/diffbot/provider/diffbot.py +++ b/diffbot/provider/diffbot.py @@ -6,13 +6,18 @@ from dify_plugin.errors.tool import ToolProviderCredentialValidationError from tools.client import DiffbotError -from tools.common import make_client +from tools.sdk import make_client, mapped_errors, verify_token class DiffbotProvider(ToolProvider): def _validate_credentials(self, credentials: dict[str, Any]) -> None: try: - make_client(credentials).validate_token() + client = make_client(credentials) + with client, mapped_errors(client): + # verify_token() never hands back the account payload (it contains the token). + valid = verify_token(client) + if not valid: + raise DiffbotError("Invalid or unauthorized Diffbot token.") except DiffbotError as exc: raise ToolProviderCredentialValidationError(str(exc)) from exc except Exception as exc: diff --git a/diffbot/tools/client.py b/diffbot/tools/client.py index d157272..6c5d13c 100644 --- a/diffbot/tools/client.py +++ b/diffbot/tools/client.py @@ -1,27 +1,15 @@ -"""HTTP client for the Diffbot APIs used by this plugin. +"""Plugin policy shared by every tool: token format, URL/host guard, safe errors. -Authentication is a Diffbot token. Extract, Knowledge Graph, Enhance, Account, -and Natural Language calls send it as the ``token`` query parameter. Web Search -on ``llm.diffbot.com`` uses ``Authorization: Bearer`` instead, which is what -the current Web Search reference documents. +HTTP itself is done by the Diffbot Python SDK (see ``tools/sdk.py``); this module +holds only the rules the plugin enforces on top of it. """ from __future__ import annotations import ipaddress import re -from typing import Any from urllib.parse import urlparse -import requests - -ACCOUNT_URL = "https://api.diffbot.com/v4/account" -EXTRACT_BASE = "https://api.diffbot.com/v3" -WEB_SEARCH_URL = "https://llm.diffbot.com/api/v1/web_search" -DQL_URL = "https://kg.diffbot.com/kg/v3/dql" -ENHANCE_URL = "https://kg.diffbot.com/kg/v3/enhance" -NLP_URL = "https://nl.diffbot.com/v1/" - TOKEN_PATTERN = re.compile(r"^[A-Za-z0-9._~+\-]{16,256}$") BLOCKED_HOST_SUFFIXES = (".local", ".localhost", ".internal") BLOCKED_HOSTS = { @@ -31,186 +19,16 @@ "metadata.google", } -TIMEOUTS = { - "account": 15, - "search": 30, - "extract": 45, - "dql": 30, - "enhance": 30, - "nlp": 30, -} - class DiffbotError(Exception): """A safe, user-facing Diffbot failure. Messages never include the token.""" -class DiffbotClient: - def __init__(self, token: str, session: requests.Session | None = None) -> None: - self.token = normalize_token(token) - self.session = session or requests.Session() - - def validate_token(self) -> None: - """Confirm the token with the Account API without returning account details. - - ``days=1`` keeps the usage history small. The response body contains the - token itself, so this method returns nothing and callers must not log it. - """ - data = self._request_json( - "GET", - ACCOUNT_URL, - params={"token": self.token, "days": 1}, - timeout=TIMEOUTS["account"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot account check returned an unexpected response.") - if not any(key in data for key in ("plan", "status", "email", "name")): - raise DiffbotError("Diffbot account check did not return account details.") - - def web_search( - self, - query: str, - *, - num_results: int, - snippet_length: str, - max_tokens: int | None, - rerank: str | None, - ) -> dict[str, Any]: - params: dict[str, Any] = { - "text": query, - "size": num_results, - "snippetLength": snippet_length.upper(), - } - if max_tokens is not None: - params["maxTokens"] = max_tokens - if rerank: - params["rerank"] = rerank - data = self._request_json( - "GET", - WEB_SEARCH_URL, - params=params, - headers={"Authorization": f"Bearer {self.token}", "Accept": "application/json"}, - timeout=TIMEOUTS["search"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot web search returned an unexpected response.") - return data - - def extract(self, url: str, *, page_type: str, markdown: bool) -> dict[str, Any]: - endpoint = "analyze" if page_type == "auto" else page_type - params: dict[str, Any] = { - "token": self.token, - "url": url, - "timeout": 30000, - } - if markdown: - params["mode"] = "llm" - data = self._request_json( - "GET", - f"{EXTRACT_BASE}/{endpoint}", - params=params, - timeout=TIMEOUTS["extract"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot extract returned an unexpected response.") - return data - - def dql(self, query: str, *, size: int) -> dict[str, Any]: - data = self._request_json( - "GET", - DQL_URL, - params={"token": self.token, "query": query, "size": size}, - timeout=TIMEOUTS["dql"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot Knowledge Graph search returned an unexpected response.") - return data - - def enhance(self, params: dict[str, Any]) -> dict[str, Any]: - data = self._request_json( - "GET", - ENHANCE_URL, - params={"token": self.token, **params}, - timeout=TIMEOUTS["enhance"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot Enhance returned an unexpected response.") - return data - - def resolve_entities(self, text: str, *, lang: str) -> dict[str, Any]: - data = self._request_json( - "POST", - NLP_URL, - params={"token": self.token, "fields": "entities,sentiment"}, - json_body=[{"lang": lang, "format": "plain text", "content": text}], - timeout=TIMEOUTS["nlp"], - ) - if isinstance(data, list): - data = data[0] if data else {} - if not isinstance(data, dict): - raise DiffbotError("Diffbot entity resolution returned an unexpected response.") - return data - - def _request_json( - self, - method: str, - url: str, - *, - params: dict[str, Any] | None = None, - json_body: Any = None, - headers: dict[str, str] | None = None, - timeout: float, - ) -> Any: - try: - response = self.session.request( - method, - url, - params=params, - json=json_body, - headers=headers, - timeout=timeout, - ) - except requests.Timeout as exc: - raise DiffbotError("The Diffbot request timed out.") from exc - except requests.RequestException as exc: - raise DiffbotError("The Diffbot request failed before a response was received.") from exc - - if response.status_code in (401, 403): - raise DiffbotError("Invalid or unauthorized Diffbot token.") - if response.status_code == 429: - raise DiffbotError("Diffbot rate limit exceeded. Retry later.") - if response.status_code >= 400: - raise DiffbotError( - f"Diffbot API error {response.status_code}: {self._safe_body_message(response)}" - ) - - try: - data = response.json() - except ValueError as exc: - raise DiffbotError("Diffbot returned a non-JSON response.") from exc - - if isinstance(data, dict) and data.get("errorCode"): - message = str(data.get("error") or data.get("message") or "request failed") - lowered = message.lower() - if "token" in lowered or "authorized" in lowered or "unauthor" in lowered: - raise DiffbotError("Invalid or unauthorized Diffbot token.") - raise DiffbotError(f"Diffbot request failed: {self._redact(message)}") - return data - - def _safe_body_message(self, response: requests.Response) -> str: - try: - data = response.json() - except ValueError: - return "request failed" - if isinstance(data, dict): - message = str(data.get("error") or data.get("message") or "request failed") - return self._redact(message)[:300] - return "request failed" - - def _redact(self, text: str) -> str: - if self.token and self.token in text: - text = text.replace(self.token, "[redacted]") - return text +def redact(text: str, token: str | None) -> str: + """Remove the token (and any ``token=`` query value) from text.""" + if token: + text = text.replace(token, "[redacted]") + return re.sub(r"(token=)[^&\s\"']+", r"\1[redacted]", text) def normalize_token(raw: str) -> str: @@ -244,7 +62,13 @@ def validate_public_http_url(url: str) -> str: raise DiffbotError("URL must start with http:// or https://.") if parsed.username or parsed.password: raise DiffbotError("URL must not include a username or password.") - host = (parsed.hostname or "").rstrip(".").lower() + check_public_host(parsed.hostname or "") + return url + + +def check_public_host(hostname: str) -> None: + """Reject empty, local-named, metadata, and private/loopback literal hosts.""" + host = hostname.rstrip(".").lower() if not host: raise DiffbotError("URL must include a host.") if host in BLOCKED_HOSTS or host.endswith(BLOCKED_HOST_SUFFIXES): @@ -252,8 +76,8 @@ def validate_public_http_url(url: str) -> str: try: ip = ipaddress.ip_address(host) except ValueError: - ip = None - if ip is not None and ( + return + if ( ip.is_private or ip.is_loopback or ip.is_link_local @@ -262,4 +86,13 @@ def validate_public_http_url(url: str) -> str: or ip.is_unspecified ): raise DiffbotError("URL host is not allowed.") - return url + + +def normalize_page_url(url: str) -> str: + """Prepend https:// to a bare domain (as the SDK does), then run the host guard.""" + if not isinstance(url, str): + raise DiffbotError("A page URL is required.") + url = url.strip() + if url and "://" not in url: + url = f"https://{url}" + return validate_public_http_url(url) diff --git a/diffbot/tools/common.py b/diffbot/tools/common.py index 8a3ae4b..a4aa702 100644 --- a/diffbot/tools/common.py +++ b/diffbot/tools/common.py @@ -7,19 +7,11 @@ from dify_plugin.entities.tool import ToolInvokeMessage -from tools.client import DiffbotClient +from tools.sdk import as_dict, make_client, mapped_errors # noqa: F401 (re-exported for tools) from tools.dql_builder import bounded_number from tools.shape import clip -def make_client(credentials: dict[str, Any] | None) -> DiffbotClient: - raw = "" - if credentials: - value = credentials.get("diffbot_token", "") - raw = value if isinstance(value, str) else "" - return DiffbotClient(raw) - - def optional_text(parameters: dict[str, Any], name: str) -> str: value = parameters.get(name) if value is None: @@ -35,6 +27,22 @@ def result_size(parameters: dict[str, Any], *, default: int, maximum: int) -> in return int(number) +def result_offset(parameters: dict[str, Any]) -> int: + value = parameters.get("offset") + if value is None or value == "": + return 0 + return int(bounded_number(value, label="offset", minimum=0, maximum=100000, integer=True)) + + +def run_dql(credentials: dict[str, Any] | None, query: str, *, size: int, offset: int) -> dict[str, Any]: + """Run a DQL query through the SDK (POST) and record the query that ran.""" + client = make_client(credentials) + with client, mapped_errors(client): + data = as_dict(client.dql(query, size=size, from_=offset), "Knowledge Graph search") + data["dql"] = query + return data + + def messages(tool: Any, summary: str, payload: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: yield tool.create_text_message(clip(summary)) yield tool.create_json_message(payload) diff --git a/diffbot/tools/enhance_entity.py b/diffbot/tools/enhance_entity.py index 983c1bd..27e0765 100644 --- a/diffbot/tools/enhance_entity.py +++ b/diffbot/tools/enhance_entity.py @@ -7,7 +7,8 @@ from dify_plugin.entities.tool import ToolInvokeMessage from tools.client import DiffbotError, validate_public_http_url -from tools.common import make_client, messages, optional_text, result_size +from tools.common import make_client, mapped_errors, messages, optional_text, result_size +from tools.sdk import enhance from tools.shape import summarize_lines, trim_enhance @@ -43,7 +44,11 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag if title: params["title"] = title - data = make_client(self.runtime.credentials).enhance(params) + # Enhance is not part of the Diffbot SDK; this goes through the SDK's transport + # (retries, timeouts, error mapping) via a plugin-side wrapper. + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + data = enhance(client, params) payload = trim_enhance(data, entity_type) lines = [] for index, match in enumerate(payload["matches"], start=1): diff --git a/diffbot/tools/extract.py b/diffbot/tools/extract.py index 0ed7a53..daf8ce4 100644 --- a/diffbot/tools/extract.py +++ b/diffbot/tools/extract.py @@ -1,21 +1,82 @@ """Diffbot page extraction tool.""" +import re from collections.abc import Generator from typing import Any from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.client import DiffbotError, validate_public_http_url -from tools.common import make_client, messages, optional_text +from tools.client import DiffbotError, normalize_page_url +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import as_dict, extract from tools.shape import summarize_lines, trim_extract PAGE_TYPES = {"auto", "article", "product", "discussion", "image", "video", "list", "event"} +FIELD_PATTERN = re.compile(r"^[A-Za-z0-9_.*,\-]{1,500}$") +# Value sets from the SDK 3.3.0 types (ExtractMode, ExtractFallback, UseProxy, NaturalLanguageField). +MODES = {"article", "product", "discussion", "image", "video", "list", "event", "llm"} +FALLBACKS = {"article", "product", "discussion", "image", "video", "list", "event"} +PROXIES = {"default", "none"} +NATURAL_LANGUAGE = { + "entities", "sentiment", "summary", "facts", "openFacts", "records", "categories", "sentences", "language", +} +_NUMBERS = ( + # name, minimum, maximum, integer + ("max_pages", 1, 100, True), + ("summary_num_sentences", 1, 20, True), + ("max_tags", 1, 100, True), + ("tag_confidence", 0, 1, False), + ("category_confidence", 0, 1, False), +) +_BOOLEANS = ("discussion", "paging") + + +def _extract_options(parameters: dict[str, Any]) -> dict[str, Any]: + """Caller-set Extract options, named as in diffbot 3.3.0; unset ones are omitted.""" + options: dict[str, Any] = {} + fields = optional_text(parameters, "fields").replace(" ", "") + if fields: + if not FIELD_PATTERN.fullmatch(fields): + raise DiffbotError("fields must be a comma-separated list of field names.") + options["fields"] = [part for part in fields.split(",") if part] + for name, allowed in (("mode", MODES), ("fallback", FALLBACKS), ("use_proxy", PROXIES)): + value = optional_text(parameters, name) + if value: + if value not in allowed: + raise DiffbotError(f"{name} must be one of: {', '.join(sorted(allowed))}.") + options[name] = value + timeout = parameters.get("timeout") + if timeout not in (None, ""): + # Milliseconds, the Extract API's unit. The plugin runtime kills calls at 300 s. + options["timeout"] = int( + bounded_number(timeout, label="timeout", minimum=1000, maximum=250000, integer=True) + ) + for name, low, high, integer in _NUMBERS: + value = parameters.get(name) + if value not in (None, ""): + number = bounded_number(value, label=name, minimum=low, maximum=high, integer=integer) + options[name] = int(number) if integer else float(number) + for name in _BOOLEANS: + value = parameters.get(name) + if isinstance(value, bool): + if value: + options[name] = True # false is the API default, so only send true + natural = [part for part in optional_text(parameters, "natural_language").replace(" ", "").split(",") if part] + if natural: + bad = [part for part in natural if part not in NATURAL_LANGUAGE] + if bad: + raise DiffbotError(f"natural_language must use: {', '.join(sorted(NATURAL_LANGUAGE))}.") + options["natural_language"] = natural + return options + + class ExtractTool(Tool): def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: - url = validate_public_http_url(optional_text(tool_parameters, "url")) + url = normalize_page_url(optional_text(tool_parameters, "url")) page_type = optional_text(tool_parameters, "page_type").lower() or "auto" if page_type not in PAGE_TYPES: raise DiffbotError( @@ -25,11 +86,13 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag if output_format not in {"markdown", "json"}: raise DiffbotError("format must be markdown or json.") - data = make_client(self.runtime.credentials).extract( - url, - page_type=page_type, - markdown=(output_format == "markdown"), - ) + options = _extract_options(tool_parameters) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + data = as_dict( + extract(client, url, "analyze" if page_type == "auto" else page_type, output_format, options), + "extract", + ) payload = trim_extract(data) payload["url"] = url payload["page_type"] = page_type diff --git a/diffbot/tools/extract.yaml b/diffbot/tools/extract.yaml index efdda6e..13a4f49 100644 --- a/diffbot/tools/extract.yaml +++ b/diffbot/tools/extract.yaml @@ -20,7 +20,7 @@ parameters: label: en_US: Page URL human_description: - en_US: Public http or https URL. Local and private hosts are rejected. + en_US: Public http or https URL; a bare domain like example.com gets https://. Local and private hosts are rejected. llm_description: Absolute public http or https URL of the page to extract. placeholder: en_US: https://example.com/story @@ -76,6 +76,191 @@ parameters: - value: json label: en_US: JSON + - name: fields + type: string + required: false + form: llm + label: + en_US: Fields + human_description: + en_US: Comma-separated extra fields to return, for example links,meta,tags. Leave empty for the defaults. + llm_description: Optional comma-separated Diffbot extract fields, for example links,meta or sentiment. + - name: timeout + type: number + required: false + form: form + min: 1000 + max: 250000 + label: + en_US: Extraction timeout (ms) + human_description: + en_US: Milliseconds Diffbot may spend on the page. Default 30000. + llm_description: Leave empty. + - name: discussion + type: boolean + required: false + form: form + default: false + label: + en_US: Include discussion + human_description: + en_US: "Article/product: also extract comments and reviews." + llm_description: Leave unset. + - name: paging + type: boolean + required: false + form: form + default: false + label: + en_US: Follow pagination + human_description: + en_US: Stitch multi-page articles together. + llm_description: Leave unset. + - name: max_pages + type: number + required: false + form: form + min: 1 + max: 100 + label: + en_US: Max pages + human_description: + en_US: With pagination on, the most pages to follow (1-100). + llm_description: Leave empty. + - name: natural_language + type: string + required: false + form: form + label: + en_US: Natural language + human_description: + en_US: "Comma-separated natural-language fields to add (article): entities, sentiment, summary, facts, openFacts, records, categories, sentences, language." + llm_description: Leave empty. + - name: summary_num_sentences + type: number + required: false + form: form + min: 1 + max: 20 + label: + en_US: Summary sentences + human_description: + en_US: Sentences in the summary when natural language includes summary (1-20). + llm_description: Leave empty. + - name: max_tags + type: number + required: false + form: form + min: 1 + max: 100 + label: + en_US: Max tags + human_description: + en_US: Most tags to return per page (1-100). + llm_description: Leave empty. + - name: tag_confidence + type: number + required: false + form: form + min: 0 + max: 1 + label: + en_US: Tag confidence + human_description: + en_US: Minimum tag confidence, 0 to 1. + llm_description: Leave empty. + - name: category_confidence + type: number + required: false + form: form + min: 0 + max: 1 + label: + en_US: Category confidence + human_description: + en_US: Minimum category confidence, 0 to 1. + llm_description: Leave empty. + - name: mode + type: select + required: false + form: form + label: + en_US: Mode + human_description: + en_US: Restrict Analyze to one Extract API, or llm for markdown. Overrides the Markdown format's default (llm). + llm_description: Leave empty. + options: + - value: article + label: + en_US: article + - value: product + label: + en_US: product + - value: discussion + label: + en_US: discussion + - value: image + label: + en_US: image + - value: video + label: + en_US: video + - value: list + label: + en_US: list + - value: event + label: + en_US: event + - value: llm + label: + en_US: llm + - name: fallback + type: select + required: false + form: form + label: + en_US: Fallback page type + human_description: + en_US: API to use when Analyze classifies the page as other. + llm_description: Leave empty. + options: + - value: article + label: + en_US: article + - value: product + label: + en_US: product + - value: discussion + label: + en_US: discussion + - value: image + label: + en_US: image + - value: video + label: + en_US: video + - value: list + label: + en_US: list + - value: event + label: + en_US: event + - name: use_proxy + type: select + required: false + form: form + label: + en_US: Proxy + human_description: + en_US: Fetch through Diffbot's datacenter proxy (default) or explicitly none. + llm_description: Leave empty. + options: + - value: default + label: + en_US: default + - value: none + label: + en_US: none extra: python: source: tools/extract.py diff --git a/diffbot/tools/resolve_entities.py b/diffbot/tools/resolve_entities.py index d5716d8..679e506 100644 --- a/diffbot/tools/resolve_entities.py +++ b/diffbot/tools/resolve_entities.py @@ -7,7 +7,7 @@ from dify_plugin.entities.tool import ToolInvokeMessage from tools.client import DiffbotError -from tools.common import make_client, messages, optional_text +from tools.common import as_dict, make_client, mapped_errors, messages, optional_text from tools.dql_builder import validate_language from tools.shape import summarize_lines, trim_resolved_entities @@ -23,7 +23,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag if lang != "auto": lang = validate_language(lang) - data = make_client(self.runtime.credentials).resolve_entities(text, lang=lang) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + data = as_dict(client.entities(text, lang=lang), "entity resolution") payload = trim_resolved_entities(data) lines = [] for entity in payload["entities"]: diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py new file mode 100644 index 0000000..8b71335 --- /dev/null +++ b/diffbot/tools/sdk.py @@ -0,0 +1,302 @@ +"""Builds the Diffbot SDK client and adapts it to this plugin's policy. + +The plugin runs on the ``diffbot`` Python SDK (sync ``Diffbot`` client). Everything +the SDK does not expose yet is reached through the public ``client.request()`` +transport, never through private attributes, and each such helper feature-detects +the real method so it drops out when the SDK ships it: + +* ``verify_token`` -> diffbot 3.3.0 (fallback: ``GET /v4/account?days=1``) +* ``crawl_download`` -> diffbot 3.3.0 (fallback: ``GET /v3/crawl/data``) +* extract options -> diffbot 3.3.0 (fallback: ``GET /v3/{api}`` with the same params) +* Enhance -> never in the SDK; a thin plugin-side wrapper stays. + +TODO: when diffbot 3.2.0 / 3.3.0 are released, pin ``diffbot`` in +``requirements.txt`` / ``pyproject.toml`` and delete the fallbacks. +""" + +from __future__ import annotations + +import csv +import inspect +import io +import json +from collections.abc import Iterator, Mapping +from contextlib import contextmanager +from typing import Any, cast + +import httpx +from diffbot import ( + APIError, + AuthError, + Diffbot, + DiffbotError as SdkError, + ExtractionError, + NotFoundError, + RateLimitError, + ValidationError, +) + +from tools.client import ( + DiffbotError, + check_public_host, + normalize_token, + redact, +) +from tools.sdk_types import CrawlJob, ExtractResponse + +DEFAULT_TIMEOUT = 30.0 +MIN_TIMEOUT = 5.0 +MAX_TIMEOUT = 300.0 # dify_plugin's request ceiling is 300 s + +INVALID_TOKEN = "Invalid or unauthorized Diffbot token." + + +class GuardedTransport(httpx.BaseTransport): + """Wraps another transport and refuses private/loopback/metadata hosts and + non-http(s) schemes before any socket is opened.""" + + def __init__(self, inner: httpx.BaseTransport | None = None, *, block_private_hosts: bool = True) -> None: + self._inner = inner or httpx.HTTPTransport() + self._block = block_private_hosts + + def handle_request(self, request: httpx.Request) -> httpx.Response: + if request.url.scheme not in ("http", "https"): + raise DiffbotError("Request scheme is not allowed.") + if self._block: + check_public_host(request.url.host or "") + return self._inner.handle_request(request) + + def close(self) -> None: + self._inner.close() + + +def _as_bool(value: Any, default: bool) -> bool: + if value is None or value == "": + return default + if isinstance(value, str): + return value.strip().lower() not in {"false", "0", "no", "off"} + return bool(value) + + +def parse_timeout(value: Any) -> float: + if value is None or (isinstance(value, str) and not value.strip()): + return DEFAULT_TIMEOUT + try: + seconds = float(value) + except (TypeError, ValueError): + raise DiffbotError("Request timeout must be a number of seconds.") from None + if not MIN_TIMEOUT <= seconds <= MAX_TIMEOUT: + raise DiffbotError(f"Request timeout must be between {MIN_TIMEOUT:g} and {MAX_TIMEOUT:g} seconds.") + return seconds + + +def make_client(credentials: Mapping[str, Any] | None, *, transport: httpx.BaseTransport | None = None) -> Diffbot: + """Validate the credential form and return an SDK client behind the host guard.""" + credentials = credentials or {} + raw = credentials.get("diffbot_token", "") + token = normalize_token(raw if isinstance(raw, str) else "") + timeout = parse_timeout(credentials.get("request_timeout")) + block = _as_bool(credentials.get("block_private_hosts"), True) + return Diffbot(token, timeout=timeout, transport=GuardedTransport(transport, block_private_hosts=block)) + + +def api_error_message(exc: BaseException, token: str) -> str: + """Safe, user-facing text for an SDK/HTTP failure. Never includes the token.""" + if isinstance(exc, DiffbotError): + return redact(str(exc), token) + if isinstance(exc, AuthError): + return INVALID_TOKEN + if isinstance(exc, RateLimitError): + wait = f" Retry after {redact(str(exc.retry_after), token)} seconds." if exc.retry_after else " Retry later." + return f"Diffbot rate limit exceeded.{wait}" + if isinstance(exc, ExtractionError): + if exc.error_code in (401, 403): + return INVALID_TOKEN + return f"Diffbot could not extract the page ({exc.error_code}): {redact(exc.error, token)[:300]}" + if isinstance(exc, NotFoundError): + return "Diffbot did not find that resource." + if isinstance(exc, APIError): + detail = redact(exc.message or "request failed", token)[:300] + return f"Diffbot API error {exc.status_code}: {detail}" + if isinstance(exc, httpx.TimeoutException): + return "The Diffbot request timed out." + if isinstance(exc, (httpx.HTTPError, SdkError)): + return "The Diffbot request failed before a response was received." + return "The Diffbot request failed." + + +@contextmanager +def mapped_errors(client: Diffbot) -> Iterator[None]: + """Translate SDK/httpx errors to plugin-safe ``DiffbotError`` messages.""" + try: + yield + except DiffbotError: + raise + except ValidationError as exc: + raise DiffbotError(redact(str(exc), client.token)) from None + except (SdkError, httpx.HTTPError) as exc: + raise DiffbotError(api_error_message(exc, client.token)) from None + + +# ---------------------------------------------------------------- credential check + + +def verify_token(client: Diffbot) -> bool: + """True if the token is accepted. Never returns the account payload (it holds the token). + + Only an auth failure means False; other errors propagate. + """ + native = getattr(client, "verify_token", None) + if callable(native): + return bool(native()) + try: + client.request("GET", f"{client.api_base}/v4/account", params={"token": client.token, "days": 1}) + except AuthError: + return False + return True + + +# ---------------------------------------------------------------- extract + +EXTRACT_OPTION_WIRE = { + "timeout": "timeout", + "mode": "mode", + "fields": "fields", + "fallback": "fallback", + "discussion": "discussion", + "paging": "paging", + "max_pages": "maxPages", + "natural_language": "naturalLanguage", + "summary_num_sentences": "summaryNumSentences", + "max_tags": "maxTags", + "tag_confidence": "tagConfidence", + "category_confidence": "categoryConfidence", + "use_proxy": "useProxy", +} +_LIST_OPTIONS = {"fields", "natural_language"} + + +def _supports_extract_options(client: Diffbot) -> bool: + try: + params = inspect.signature(client.extract).parameters + except (TypeError, ValueError): + return False + return "mode" in params or any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()) + + +def _wire_value(value: Any) -> Any: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (list, tuple)): + return ",".join(str(item) for item in value) + return value + + +def extract(client: Diffbot, url: str, api: str, fmt: str, options: Mapping[str, Any]) -> ExtractResponse: + """Extract ``url``. ``options`` use the SDK 3.3.0 keyword names (unset keys omitted).""" + options = {key: value for key, value in options.items() if value is not None} + if not options or _supports_extract_options(client): + return cast(ExtractResponse, client.extract(url, api, fmt, **options)) + # Fallback for SDKs without extract options: same request the SDK builds. + params: dict[str, Any] = {"token": client.token, "url": url, "timeout": 30000} + if fmt == "markdown": + params["mode"] = "llm" + for key, value in options.items(): + params[EXTRACT_OPTION_WIRE[key]] = _wire_value(value) + timeout = params.get("timeout") + per_call = (float(timeout) / 1000.0 + 10.0) if isinstance(timeout, (int, float)) else None + response = client.request("GET", f"{client.analyze_url}/{api}", params=params, timeout=per_call) + data = response.json() + if isinstance(data, dict) and "errorCode" in data: + raise ExtractionError(data["errorCode"], data.get("error", "")) + return cast(ExtractResponse, data) + + +# ---------------------------------------------------------------- crawl + + +def crawl_start(client: Diffbot, seeds: str, **kwargs: Any) -> str: + """Create a crawl job (never watches) and return its name.""" + for event in client.crawl(seeds, watch=False, **kwargs): + name = (event.details or {}).get("job_name") + if name: + return str(name) + raise DiffbotError("Diffbot did not report a crawl job name.") + + +def crawl_get_job(client: Diffbot, job_name: str) -> CrawlJob: + job = client.crawl_get_job(job_name) + if not job: + raise DiffbotError("Diffbot did not find that crawl job.") + return cast(CrawlJob, job) + + +def _download_text(result: Any) -> Any: + """Coerce a ``crawl_download`` return value (Response, bytes, str, parsed) to text/objects.""" + if isinstance(result, (list, dict)): + return result + if hasattr(result, "text"): + return result.text + if isinstance(result, (bytes, bytearray)): + return bytes(result).decode("utf-8", errors="replace") + return str(result) + + +def crawl_download( + client: Diffbot, job_name: str, *, format: str = "json", type: str | None = None, num: int | None = None +) -> Any: + """Fetch crawl output as parsed rows/objects (a list, or a dict for JSON objects).""" + if type == "urls": + format = "csv" # the URL report is CSV; the JSON decoder would reject it + native = getattr(client, "crawl_download", None) + if callable(native): + raw = native(job_name, format=format, type=type, num=num) + else: + params: dict[str, Any] = {"token": client.token, "name": job_name} + if format and format != "json": + params["format"] = format + if type: + params["type"] = type + if num is not None: + params["num"] = num + raw = client.request("GET", f"{client.crawler_url}/data", params=params, follow_redirects=True) + data = _download_text(raw) + if not isinstance(data, str): + return data + if not data.strip(): + return [] + try: + return json.loads(data) + except ValueError: + return list(csv.DictReader(io.StringIO(data))) + + +# ---------------------------------------------------------------- enhance (plugin-side only) + + +def enhance(client: Diffbot, params: Mapping[str, Any]) -> dict[str, Any]: + """Thin wrapper over the public transport; the SDK deliberately has no Enhance.""" + response = client.request("GET", f"{client.kg_base}/kg/v3/enhance", params={"token": client.token, **params}) + data = response.json() + if not isinstance(data, dict): + raise DiffbotError("Diffbot Enhance returned an unexpected response.") + return data + + +def as_dict(value: Any, what: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise DiffbotError(f"Diffbot {what} returned an unexpected response.") + return value + + +__all__ = [ + "GuardedTransport", + "make_client", + "mapped_errors", + "verify_token", + "extract", + "crawl_start", + "crawl_get_job", + "crawl_download", + "enhance", +] diff --git a/diffbot/tools/sdk_types.py b/diffbot/tools/sdk_types.py new file mode 100644 index 0000000..db3f5be --- /dev/null +++ b/diffbot/tools/sdk_types.py @@ -0,0 +1,63 @@ +"""Response shapes the plugin reads from the Diffbot SDK. + +TODO(diffbot 3.3.0): the SDK will ship response TypedDicts (``AccountResponse`` and +friends). Until it does these local, ``total=False`` equivalents describe only the +fields this plugin touches. Swap them for ``from diffbot.types import ...`` then. +""" + +from __future__ import annotations + +from typing import Any, TypedDict + +try: # diffbot >= 3.3.0 ships these; prefer them. + from diffbot.types import ( # type: ignore[attr-defined] + CrawlJob, + CrawlJobStatus, + DqlResponse, + EntitiesResponse as NlpResponse, + ExtractResponse, + WebSearchResponse, + ) + + SDK_TYPES = True +except ImportError: # diffbot 3.2.x: local equivalents below + SDK_TYPES = False + + +if not SDK_TYPES: + class ExtractResponse(TypedDict, total=False): + objects: list[dict[str, Any]] + humanLanguage: str + errorCode: int + error: str + + + class WebSearchResponse(TypedDict, total=False): + query: list[str] + timeMs: int + search_results: list[dict[str, Any]] + + + class DqlResponse(TypedDict, total=False): + hits: int + data: list[dict[str, Any]] + dql: str + + + class NlpResponse(TypedDict, total=False): + entities: list[dict[str, Any]] + sentiment: float + language: str + + + class CrawlJobStatus(TypedDict, total=False): + status: int + message: str + + + class CrawlJob(TypedDict, total=False): + name: str + jobStatus: CrawlJobStatus + pageCrawlSuccesses: int + pageProcessSuccesses: int + objectsFound: int diff --git a/diffbot/tools/search_news.py b/diffbot/tools/search_news.py index 7cdf6c7..ccbdae9 100644 --- a/diffbot/tools/search_news.py +++ b/diffbot/tools/search_news.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_news_query from tools.shape import summarize_lines, trim_kg @@ -23,11 +23,10 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "until": optional_text(tool_parameters, "until"), } query = build_news_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) + size = result_size(tool_parameters, default=5, maximum=100) include_text = bool(tool_parameters.get("include_text")) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "article", include_text=include_text) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "article", include_text=include_text, limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): headline = row.get("title") or "Untitled article" diff --git a/diffbot/tools/search_news.yaml b/diffbot/tools/search_news.yaml index ea1249c..9893f90 100644 --- a/diffbot/tools/search_news.yaml +++ b/diffbot/tools/search_news.yaml @@ -92,12 +92,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many articles to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many articles to return, from 1 to 100. + llm_description: Result cap from 1 to 1000. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_news.py diff --git a/diffbot/tools/search_organizations.py b/diffbot/tools/search_organizations.py index db1a113..55a41b0 100644 --- a/diffbot/tools/search_organizations.py +++ b/diffbot/tools/search_organizations.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_organizations_query from tools.shape import summarize_lines, trim_kg @@ -26,10 +26,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "is_public": optional_text(tool_parameters, "is_public").lower() or "any", } query = build_organizations_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "organization") + size = result_size(tool_parameters, default=5, maximum=100) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "organization", limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): location = row.get("location") or {} diff --git a/diffbot/tools/search_organizations.yaml b/diffbot/tools/search_organizations.yaml index d6ef184..3b43400 100644 --- a/diffbot/tools/search_organizations.yaml +++ b/diffbot/tools/search_organizations.yaml @@ -123,12 +123,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many organizations to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many organizations to return, from 1 to 100. + llm_description: Result cap from 1 to 1000. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_organizations.py diff --git a/diffbot/tools/search_people.py b/diffbot/tools/search_people.py index 8bf281f..44272df 100644 --- a/diffbot/tools/search_people.py +++ b/diffbot/tools/search_people.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_people_query from tools.shape import summarize_lines, trim_kg @@ -21,10 +21,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "country": optional_text(tool_parameters, "country"), } query = build_people_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "person") + size = result_size(tool_parameters, default=5, maximum=100) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "person", limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): jobs = row.get("employments") or [] diff --git a/diffbot/tools/search_people.yaml b/diffbot/tools/search_people.yaml index 2f3c388..134ab83 100644 --- a/diffbot/tools/search_people.yaml +++ b/diffbot/tools/search_people.yaml @@ -64,12 +64,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many people to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many people to return, from 1 to 100. + llm_description: Result cap from 1 to 1000. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_people.py diff --git a/diffbot/tools/search_web.py b/diffbot/tools/search_web.py index aff70ef..1616bb7 100644 --- a/diffbot/tools/search_web.py +++ b/diffbot/tools/search_web.py @@ -7,7 +7,7 @@ from dify_plugin.entities.tool import ToolInvokeMessage from tools.client import DiffbotError -from tools.common import make_client, messages, optional_text +from tools.common import as_dict, make_client, mapped_errors, messages, optional_text from tools.dql_builder import bounded_number from tools.shape import summarize_lines, trim_web_search @@ -32,32 +32,34 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag num_results = 5 else: num_results = int( - bounded_number(num_value, label="num_results", minimum=1, maximum=10, integer=True) + bounded_number(num_value, label="num_results", minimum=1, maximum=50, integer=True) ) snippet = optional_text(tool_parameters, "snippet_length").lower() - rerank = None - if not snippet: - snippet = "full" if _is_page_lookup(query) else "short" - if snippet not in SNIPPETS: + if not snippet and _is_page_lookup(query): + snippet = "full" # a lone url: query asks for the stored page + if snippet and snippet not in SNIPPETS: raise DiffbotError("snippet_length must be short, medium, long, or full.") - if snippet == "full": - rerank = "none" max_tokens = None raw_tokens = tool_parameters.get("max_tokens") if raw_tokens not in (None, ""): max_tokens = int( - bounded_number(raw_tokens, label="max_tokens", minimum=500, maximum=8000, integer=True) + bounded_number(raw_tokens, label="max_tokens", minimum=500, maximum=32000, integer=True) ) - data = make_client(self.runtime.credentials).web_search( - query, - num_results=num_results, - snippet_length=snippet, - max_tokens=max_tokens, - rerank=rerank, - ) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + # Blank snippet_length is omitted so the API default (long) applies. + data = as_dict( + client.web_search( + query, + num_results=num_results, + max_tokens=max_tokens, + snippet_length=snippet or None, # type: ignore[arg-type] + ), + "web search", + ) payload = trim_web_search(data) lines = [] for index, item in enumerate(payload["results"], start=1): diff --git a/diffbot/tools/search_web.yaml b/diffbot/tools/search_web.yaml index 9bb6a6b..90a5b63 100644 --- a/diffbot/tools/search_web.yaml +++ b/diffbot/tools/search_web.yaml @@ -32,12 +32,12 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 50 label: en_US: Number of results human_description: - en_US: How many results to return, from 1 to 10. Defaults to 5. - llm_description: Maximum number of results, from 1 to 10. Default 5. + en_US: How many results to return, from 1 to 50. Defaults to 5. + llm_description: Maximum number of results, from 1 to 50. Default 5. - name: snippet_length type: select required: false @@ -45,10 +45,10 @@ parameters: label: en_US: Snippet length human_description: - en_US: "How much of each page to return. Leave empty for a short passage, or full for the whole indexed page on a url: lookup." + en_US: "How much of each page to return. Leave empty for the Diffbot default (long). A lone url: query returns the full indexed page unless you choose a size." llm_description: > Passage size. short is about 1 KB, then medium, long, and full. - Leave empty to use short, or full automatically when the query is only a url: lookup. + Leave empty for the API default (long), or full automatically when the query is only a url: lookup. options: - value: short label: @@ -67,12 +67,12 @@ parameters: required: false form: llm min: 500 - max: 8000 + max: 32000 label: en_US: Max tokens human_description: en_US: Optional cap on the response size. Diffbot shortens passages to fit. Ignored for a full-page lookup. - llm_description: Optional response token cap from 500 to 8000. Omit it unless the context window is tight. + llm_description: Optional response token cap from 500 to 32000. Omit it unless the context window is tight. extra: python: source: tools/search_web.py diff --git a/diffbot/tools/shape.py b/diffbot/tools/shape.py index 4d43168..c6384f1 100644 --- a/diffbot/tools/shape.py +++ b/diffbot/tools/shape.py @@ -115,9 +115,9 @@ def trim_article(entity: dict[str, Any], *, include_text: bool) -> dict[str, Any return {key: value for key, value in record.items() if value not in (None, "", [], {})} -def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False) -> dict[str, Any]: +def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False, limit: int = 20) -> dict[str, Any]: rows = [] - for item in (data.get("data") or [])[:20]: + for item in (data.get("data") or [])[:limit]: entity = item.get("entity") if isinstance(item, dict) and "entity" in item else item if not isinstance(entity, dict): continue diff --git a/tests/test_tools_http.py b/tests/test_tools_http.py index cc946d4..28893d1 100644 --- a/tests/test_tools_http.py +++ b/tests/test_tools_http.py @@ -2,8 +2,8 @@ import json +import httpx import pytest -import requests from provider.diffbot import DiffbotProvider from tools.client import TOKEN_PATTERN, DiffbotError @@ -20,38 +20,38 @@ class FakeResponse: - def __init__(self, status_code, payload): - self.status_code = status_code - self._payload = payload - self.text = payload if isinstance(payload, str) else json.dumps(payload) - - def json(self): - if isinstance(self._payload, str): - raise ValueError("not json") - return self._payload - - -class FakeSession: - def __init__(self, responder): - self.responder = responder - self.calls = [] + """(status, payload) pair a responder returns; converted to an httpx.Response.""" - def request(self, method, url, **kwargs): - self.calls.append({"method": method, "url": url, **kwargs}) - return self.responder(method, url, kwargs) + def __init__(self, status_code, payload, headers=None): + self.status_code = status_code + self.payload = payload + self.headers = headers or {} @pytest.fixture def session_factory(monkeypatch): - holder = {} + """Serve every SDK request from ``responder(method, url, kwargs)`` via httpx.MockTransport. - def install(responder): - def factory(): - session = FakeSession(responder) - holder["session"] = session - return session + ``kwargs`` has ``params`` (query dict), ``headers`` and ``json`` (parsed body), so the + real SDK client and the plugin's host guard run; only the socket is replaced. + """ + holder = {"calls": []} - monkeypatch.setattr("tools.client.requests.Session", factory) + def install(responder): + def handler(request: httpx.Request) -> httpx.Response: + kwargs = { + "params": dict(request.url.params), + "headers": dict(request.headers), + "json": json.loads(request.content) if request.content else None, + } + url = str(request.url.copy_with(query=None)) + holder["calls"].append({"method": request.method, "url": url, **kwargs}) + result = responder(request.method, url, kwargs) + if isinstance(result.payload, str): + return httpx.Response(result.status_code, text=result.payload, headers=result.headers) + return httpx.Response(result.status_code, json=result.payload, headers=result.headers) + + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda: httpx.MockTransport(handler)) return holder return install @@ -70,7 +70,7 @@ def _text(messages): def test_credential_validation_uses_account_api_and_hides_the_token(session_factory): def responder(method, url, kwargs): assert kwargs["params"]["token"] == TOKEN - assert kwargs["params"]["days"] == 1 + assert kwargs["params"]["days"] == "1" return FakeResponse(200, {"plan": "FREE", "status": "active", "token": TOKEN, "email": "hidden@example.com"}) session_factory(responder) @@ -91,10 +91,10 @@ def responder(method, url, kwargs): def test_web_search_uses_bearer_auth_and_not_the_query_token(session_factory): def responder(method, url, kwargs): assert url.endswith("/api/v1/web_search") - assert kwargs["headers"]["Authorization"] == f"Bearer {TOKEN}" + assert kwargs["headers"]["authorization"] == f"Bearer {TOKEN}" assert "token" not in (kwargs.get("params") or {}) assert kwargs["params"]["text"] == "hypothyroidism" - assert kwargs["params"]["snippetLength"] == "SHORT" + assert "snippetLength" not in kwargs["params"] # API default applies return FakeResponse( 200, { @@ -173,8 +173,8 @@ def responder(method, url, kwargs): def test_organization_search_drops_unexpected_contact_fields(session_factory): def responder(method, url, kwargs): assert "kg.diffbot.com" in url - assert "type:Organization" in kwargs["params"]["query"] - assert 'name:"Diffbot"' in kwargs["params"]["query"] + assert "type:Organization" in kwargs["json"]["query"] + assert 'name:"Diffbot"' in kwargs["json"]["query"] return FakeResponse( 200, { @@ -208,7 +208,7 @@ def responder(method, url, kwargs): def test_people_search_keeps_employment_and_drops_email(session_factory): def responder(method, url, kwargs): - assert "type:Person" in kwargs["params"]["query"] + assert "type:Person" in kwargs["json"]["query"] return FakeResponse( 200, { @@ -241,7 +241,7 @@ def test_news_search_passes_the_built_query(session_factory): seen = {} def responder(method, url, kwargs): - seen["query"] = kwargs["params"]["query"] + seen["query"] = kwargs["json"]["query"] return FakeResponse( 200, { @@ -345,7 +345,7 @@ def responder(method, url, kwargs): def test_timeout_becomes_a_safe_error(session_factory): def responder(method, url, kwargs): - raise requests.Timeout("timed out with token " + TOKEN) + raise httpx.ReadTimeout("timed out with token " + TOKEN) session_factory(responder) tool = SearchWebTool.from_credentials({"diffbot_token": TOKEN}) From 5f2249fac6018b6c9b11c826373dae902fd7e9c1 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 12:58:48 -1000 Subject: [PATCH 03/15] Add crawl_start and crawl_jobs tools and credential timeout/host-guard fields crawl_start never watches and returns the job name. crawl_jobs lists, shows status, fetches results via crawl_download (or the /v3/crawl/data fallback), and deletes (labelled destructive, never retried). Provider gains request_timeout and block_private_hosts. test_sdk_wiring runs against the SDK with and without the 3.3.0 methods. Co-Authored-By: Claude Sonnet 5.5 --- diffbot/provider/diffbot.yaml | 20 +++ diffbot/tools/crawl_jobs.py | 79 +++++++++ diffbot/tools/crawl_jobs.yaml | 73 ++++++++ diffbot/tools/crawl_start.py | 78 +++++++++ diffbot/tools/crawl_start.yaml | 148 ++++++++++++++++ tests/test_sdk_wiring.py | 303 +++++++++++++++++++++++++++++++++ 6 files changed, 701 insertions(+) create mode 100644 diffbot/tools/crawl_jobs.py create mode 100644 diffbot/tools/crawl_jobs.yaml create mode 100644 diffbot/tools/crawl_start.py create mode 100644 diffbot/tools/crawl_start.yaml create mode 100644 tests/test_sdk_wiring.py diff --git a/diffbot/provider/diffbot.yaml b/diffbot/provider/diffbot.yaml index 8ddce53..9294c84 100644 --- a/diffbot/provider/diffbot.yaml +++ b/diffbot/provider/diffbot.yaml @@ -21,12 +21,32 @@ credentials_for_provider: help: en_US: A free token is available at app.diffbot.com/get-started. Dify stores it as a secret and the plugin sends it only to Diffbot. url: https://app.diffbot.com/get-started + request_timeout: + type: text-input + required: false + default: "30" + label: + en_US: Request timeout (seconds) + placeholder: + en_US: "30" + help: + en_US: Seconds to wait for each Diffbot call, from 5 to 300. Leave blank for 30. + block_private_hosts: + type: boolean + required: false + default: true + label: + en_US: Block private hosts + help: + en_US: Refuse requests to loopback, private, and cloud-metadata addresses. Leave on unless you run a self-hosted gateway. tools: - tools/search_web.yaml - tools/extract.yaml - tools/search_organizations.yaml - tools/search_people.yaml - tools/search_news.yaml + - tools/crawl_start.yaml + - tools/crawl_jobs.yaml - tools/enhance_entity.yaml - tools/resolve_entities.yaml extra: diff --git a/diffbot/tools/crawl_jobs.py b/diffbot/tools/crawl_jobs.py new file mode 100644 index 0000000..e5f241e --- /dev/null +++ b/diffbot/tools/crawl_jobs.py @@ -0,0 +1,79 @@ +"""List, inspect, fetch results of, and delete Diffbot crawl jobs.""" + +from collections.abc import Generator +from typing import Any + +from dify_plugin import Tool +from dify_plugin.entities.tool import ToolInvokeMessage + +from tools.client import DiffbotError +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import crawl_download, crawl_get_job +from tools.shape import clip, summarize_lines + +ACTIONS = {"list", "status", "results", "delete"} +RESULT_LIMIT = 1000 +JOB_FIELDS = ("name", "jobStatus", "pageCrawlAttempts", "pageCrawlSuccesses", "pageProcessAttempts", + "pageProcessSuccesses", "objectsFound", "urlsFound", "maxToCrawl", "maxToProcess") + + +def _job_summary(job: dict[str, Any]) -> dict[str, Any]: + out = {key: job[key] for key in JOB_FIELDS if key in job} + status = out.get("jobStatus") + if isinstance(status, dict): + out["jobStatus"] = {k: status[k] for k in ("status", "message") if k in status} + return out + + +def _cell(value: Any) -> Any: + return clip(value, 500) if isinstance(value, str) else value + + +class CrawlJobsTool(Tool): + def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: + action = optional_text(tool_parameters, "action").lower() + if action not in ACTIONS: + raise DiffbotError("action must be list, status, results, or delete.") + job_name = optional_text(tool_parameters, "job_name") + if action != "list" and not job_name: + raise DiffbotError("job_name is required for this action.") + + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + if action == "list": + jobs = [_job_summary(job) for job in client.crawl_list_jobs()[:100]] + lines = [f"- {job.get('name')}: {(job.get('jobStatus') or {}).get('message', '')}" for job in jobs] + yield from messages( + self, summarize_lines(f"{len(jobs)} crawl job(s).", lines), {"jobs": jobs} + ) + elif action == "status": + job = _job_summary(dict(crawl_get_job(client, job_name))) + message = (job.get("jobStatus") or {}).get("message", "") + yield from messages(self, f"Crawl job {job_name}: {message}".strip(), {"job": job}) + elif action == "results": + limit = int( + bounded_number( + tool_parameters.get("limit") or RESULT_LIMIT, + label="limit", minimum=1, maximum=RESULT_LIMIT, integer=True, + ) + ) + kind = optional_text(tool_parameters, "result_type").lower() + if kind not in ("", "urls"): + raise DiffbotError("result_type must be empty or urls.") + data = crawl_download(client, job_name, format="json", type=kind or None, num=limit) + rows = data if isinstance(data, list) else [data] + kept = [{k: _cell(v) for k, v in row.items()} if isinstance(row, dict) else row for row in rows[:limit]] + payload = { + "job_name": job_name, + "returned": len(kept), + "truncated": len(rows) > limit, + "results": kept, + } + yield from messages( + self, f"Crawl job {job_name} returned {len(kept)} row(s)" + + (" (truncated)." if payload["truncated"] else "."), payload + ) + else: # delete + client.crawl_delete_job(job_name) + yield from messages(self, f"Deleted crawl job {job_name} and its data.", {"deleted": job_name}) diff --git a/diffbot/tools/crawl_jobs.yaml b/diffbot/tools/crawl_jobs.yaml new file mode 100644 index 0000000..9b56f1c --- /dev/null +++ b/diffbot/tools/crawl_jobs.yaml @@ -0,0 +1,73 @@ +identity: + name: crawl_jobs + author: diffbot + label: + en_US: Crawl jobs +description: + human: + en_US: "List Diffbot crawl jobs, check one's status, fetch its results, or delete it. Delete is destructive: it permanently removes the job and its collected data." + llm: > + Manage Diffbot crawl jobs. action list shows all jobs; status shows one job's progress; + results returns up to 1000 rows (truncated) of its output; delete PERMANENTLY removes the job and its data, + so only use delete when the user explicitly asks for it. job_name is required except for list. +parameters: + - name: action + type: select + required: true + form: llm + default: list + label: + en_US: Action + human_description: + en_US: List, status, results, or delete. DELETE IS DESTRUCTIVE and permanently removes the job and its data. + llm_description: list, status, results, or delete (destructive, irreversible). + options: + - value: list + label: + en_US: List jobs + - value: status + label: + en_US: Status + - value: results + label: + en_US: Results + - value: delete + label: + en_US: Delete (destructive) + - name: job_name + type: string + required: false + form: llm + label: + en_US: Job name + human_description: + en_US: Name of the crawl job. Required for status, results, and delete. + llm_description: Name of the crawl job. Required for status, results, and delete. + - name: result_type + type: select + required: false + form: llm + label: + en_US: Result type + human_description: + en_US: "Results only: extracted objects (default) or the crawled URL report." + llm_description: Leave empty for extracted objects, or urls for the URL report. + options: + - value: urls + label: + en_US: URL report + - name: limit + type: number + required: false + form: form + default: 100 + min: 1 + max: 1000 + label: + en_US: Result limit + human_description: + en_US: "Results only: most rows to return (1-1000). Larger results are truncated." + llm_description: "Results only: most rows to return (1-1000). Larger results are truncated." +extra: + python: + source: tools/crawl_jobs.py diff --git a/diffbot/tools/crawl_start.py b/diffbot/tools/crawl_start.py new file mode 100644 index 0000000..cc6517c --- /dev/null +++ b/diffbot/tools/crawl_start.py @@ -0,0 +1,78 @@ +"""Start a Diffbot crawl job. Returns the job name, never results.""" + +from collections.abc import Generator +from typing import Any + +from dify_plugin import Tool +from dify_plugin.entities.tool import ToolInvokeMessage + +from tools.client import DiffbotError, validate_public_http_url +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import crawl_start + + +def _int(parameters: dict[str, Any], name: str, *, minimum: int, maximum: int) -> int | None: + value = parameters.get(name) + if value in (None, ""): + return None + return int(bounded_number(value, label=name, minimum=minimum, maximum=maximum, integer=True)) + + +def _seeds(raw: str) -> str: + parts = raw.split() + if not parts: + raise DiffbotError("At least one seed URL is required.") + if len(parts) > 20: + raise DiffbotError("Too many seed URLs (20 maximum).") + checked = [] + for part in parts: + checked.append(validate_public_http_url(part if "://" in part else f"https://{part}")) + return " ".join(checked) + + +class CrawlStartTool(Tool): + def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: + seeds = _seeds(optional_text(tool_parameters, "seeds")) + kwargs: dict[str, Any] = {} + name = optional_text(tool_parameters, "job_name") + if name: + if not all(c.isalnum() or c in "-_." for c in name) or len(name) > 100: + raise DiffbotError("job_name may use letters, digits, '-', '_' and '.' (100 characters maximum).") + kwargs["job_name"] = name + for param, key, low, high in ( + ("max_to_crawl", "max_to_crawl", 1, 10000), + ("max_to_process", "max_to_process", 1, 10000), + ("hops", "hops", 0, 10), + ): + value = _int(tool_parameters, param, minimum=low, maximum=high) + if value is not None: + kwargs[key] = value + for param in ("restrict_domain", "obey_robots", "use_proxies"): + if isinstance(tool_parameters.get(param), bool): + kwargs[param] = tool_parameters[param] + delay = tool_parameters.get("crawl_delay") + if delay not in (None, ""): + kwargs["crawl_delay"] = float(bounded_number(delay, label="crawl_delay", minimum=-1, maximum=3600)) + for param in ("url_crawl_pattern", "url_process_pattern", "custom_headers"): + value = optional_text(tool_parameters, param) + if value: + kwargs[param] = value + api_url = optional_text(tool_parameters, "api_url") + if api_url: + kwargs["api_url"] = validate_public_http_url(api_url) + + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + # watch=False always: polling would outlive the plugin's 300 s call limit. + job_name = crawl_start(client, seeds, **kwargs) + payload = { + "job_name": job_name, + "seeds": seeds.split(), + "next": "Use Crawl jobs with action status, then results, with this job_name.", + } + yield from messages( + self, + f"Started Diffbot crawl job {job_name}. It runs in the background; check it with the Crawl jobs tool.", + payload, + ) diff --git a/diffbot/tools/crawl_start.yaml b/diffbot/tools/crawl_start.yaml new file mode 100644 index 0000000..c3074ea --- /dev/null +++ b/diffbot/tools/crawl_start.yaml @@ -0,0 +1,148 @@ +identity: + name: crawl_start + author: diffbot + label: + en_US: Start crawl +description: + human: + en_US: Start a Diffbot crawl of one or more sites. This returns a job name immediately; it does not wait for pages. Check progress and fetch results with the Crawl jobs tool. + llm: > + Start a Diffbot crawl job on public http(s) seed URLs. Returns a job_name only, never results, + because crawls run in the background. Afterwards call crawl_jobs with action status until the job + is done, then action results. Do not use for private or local hosts. +parameters: + - name: seeds + type: string + required: true + form: llm + label: + en_US: Seed URLs + human_description: + en_US: One or more public URLs to start from, separated by spaces. Bare domains get https://. + llm_description: Space-separated public http(s) URLs to crawl. + - name: job_name + type: string + required: false + form: llm + label: + en_US: Job name + human_description: + en_US: Optional name (letters, digits, - _ .). Defaults to crawl-. + llm_description: Optional job name. Leave empty to auto-generate. + - name: max_to_crawl + type: number + required: false + form: form + default: 100 + min: 1 + max: 10000 + label: + en_US: Max pages to crawl + human_description: + en_US: Most pages to fetch (1-10000). + llm_description: Most pages to fetch (1-10000). + - name: max_to_process + type: number + required: false + form: form + default: 100 + min: 1 + max: 10000 + label: + en_US: Max pages to process + human_description: + en_US: Most pages to extract (1-10000). + llm_description: Most pages to extract (1-10000). + - name: hops + type: number + required: false + form: form + default: 2 + min: 0 + max: 10 + label: + en_US: Link hops + human_description: + en_US: How many links deep to follow from the seeds (0-10). + llm_description: How many links deep to follow from the seeds (0-10). + - name: restrict_domain + type: boolean + required: false + form: form + default: true + label: + en_US: Stay on seed domains + human_description: + en_US: Only follow links on the seed domains. + llm_description: Only follow links on the seed domains. + - name: url_crawl_pattern + type: string + required: false + form: form + label: + en_US: URL crawl pattern + human_description: + en_US: Only follow URLs containing one of these strings (Diffbot pattern syntax). + llm_description: Only follow URLs containing one of these strings (Diffbot pattern syntax). + - name: url_process_pattern + type: string + required: false + form: form + label: + en_US: URL process pattern + human_description: + en_US: Only extract URLs containing one of these strings. + llm_description: Only extract URLs containing one of these strings. + - name: crawl_delay + type: number + required: false + form: form + default: -1 + min: -1 + max: 3600 + label: + en_US: Crawl delay (s) + human_description: + en_US: Seconds between requests; -1 uses the Diffbot default. + llm_description: Seconds between requests; -1 uses the Diffbot default. + - name: obey_robots + type: boolean + required: false + form: form + default: false + label: + en_US: Obey robots.txt + human_description: + en_US: Respect robots.txt. + llm_description: Respect robots.txt. + - name: use_proxies + type: boolean + required: false + form: form + default: false + label: + en_US: Use proxies + human_description: + en_US: Route requests through Diffbot proxies. + llm_description: Route requests through Diffbot proxies. + - name: api_url + type: string + required: false + form: form + label: + en_US: Extraction API URL + human_description: + en_US: Optional Diffbot API URL to run on each page (public https only). + llm_description: Optional Diffbot API URL to run on each page (public https only). + - name: custom_headers + type: string + required: false + form: form + label: + en_US: Custom headers + human_description: + en_US: "Optional header to send with each request. Do not put secrets here: the crawler sends it to the sites." + llm_description: "Optional header to send with each request. Do not put secrets here: the crawler sends it to the sites." +extra: + python: + source: tools/crawl_start.py diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py new file mode 100644 index 0000000..cd650b5 --- /dev/null +++ b/tests/test_sdk_wiring.py @@ -0,0 +1,303 @@ +"""SDK wiring: credential check, host guard, extract options, crawl tools, feature detection. + +All HTTP is served by httpx.MockTransport; no live Diffbot calls. +""" + +import json + +import httpx +import pytest +import yaml +from diffbot import Diffbot + +from provider.diffbot import DiffbotProvider +from tools import sdk +from tools.client import DiffbotError +from tools.crawl_jobs import CrawlJobsTool +from tools.crawl_start import CrawlStartTool +from tools.extract import ExtractTool +from tools.search_organizations import SearchOrganizationsTool +from tools.search_web import SearchWebTool + +TOKEN = "a" * 32 +CREDS = {"diffbot_token": TOKEN} + + +@pytest.fixture(autouse=True, params=["native", "legacy"]) +def sdk_flavor(request, monkeypatch): + """Run every test against the SDK as installed (3.3.0 methods) and against an SDK + without them, which exercises the plugin's client.request() fallbacks.""" + if request.param == "legacy": + monkeypatch.delattr(Diffbot, "verify_token") + monkeypatch.delattr(Diffbot, "crawl_download") + real_extract = Diffbot.extract + + def old_extract(self, url, api="analyze", fmt="markdown"): + return real_extract(self, url, api, fmt) + + monkeypatch.setattr(Diffbot, "extract", old_extract) + return request.param + + +@pytest.fixture +def serve(monkeypatch): + """Install ``handler(request) -> httpx.Response``; returns the list of seen requests.""" + seen: list[httpx.Request] = [] + + def install(handler): + def wrapped(request): + seen.append(request) + return handler(request) + + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda: httpx.MockTransport(wrapped)) + return seen + + return install + + +def _json(messages): + return [m for m in messages if m.type.value == "json"][-1].message.json_object + + +# ------------------------------------------------------------ credentials + + +def test_validation_calls_account_with_days_1_and_never_surfaces_payload(serve): + seen = serve(lambda r: httpx.Response(200, json={"token": TOKEN, "plan": "FREE"})) + DiffbotProvider()._validate_credentials(CREDS) + assert seen[0].url.path == "/v4/account" + assert seen[0].url.params["days"] == "1" + + +def test_validation_rejects_auth_failure_without_echoing_token(serve): + serve(lambda r: httpx.Response(401, json={"error": f"bad {TOKEN}"})) + with pytest.raises(Exception, match="Invalid or unauthorized") as caught: + DiffbotProvider()._validate_credentials(CREDS) + assert TOKEN not in str(caught.value) + + +def test_validation_does_not_treat_server_errors_as_a_bad_token(serve): + serve(lambda r: httpx.Response(400, json={"message": "bad request"})) + with pytest.raises(Exception) as caught: + DiffbotProvider()._validate_credentials(CREDS) + assert "Invalid or unauthorized" not in str(caught.value) + + +@pytest.mark.parametrize("value", ["abc", "4", "301", "-1"]) +def test_request_timeout_is_validated(value): + with pytest.raises(DiffbotError, match="timeout"): + sdk.make_client({**CREDS, "request_timeout": value}) + + +def test_request_timeout_is_applied(): + client = sdk.make_client({**CREDS, "request_timeout": "45"}) + assert client._http.timeout.read == 45.0 # test-only peek at the httpx client + + +def test_native_verify_token_is_preferred(): + class Fake: + def verify_token(self): + return False + + assert sdk.verify_token(Fake()) is False # type: ignore[arg-type] + + +# ------------------------------------------------------------ host guard + + +def test_transport_blocks_private_hosts_before_any_socket(): + inner_calls = [] + inner = httpx.MockTransport(lambda r: inner_calls.append(r) or httpx.Response(200, json={})) + client = Diffbot(TOKEN, analyze_url="http://169.254.169.254/v3", transport=sdk.GuardedTransport(inner)) + with pytest.raises(DiffbotError, match="not allowed"): + client.extract("https://example.org") + assert inner_calls == [] + + +def test_guard_can_be_disabled_for_self_hosted_gateways(): + inner = httpx.MockTransport(lambda r: httpx.Response(200, json={"objects": []})) + client = Diffbot( + TOKEN, analyze_url="http://10.0.0.5/v3", transport=sdk.GuardedTransport(inner, block_private_hosts=False) + ) + assert client.extract("https://example.org") == {"objects": []} + + +def test_rate_limit_maps_to_a_safe_message(serve): + serve(lambda r: httpx.Response(429, json={"message": TOKEN}, headers={"retry-after": "0"})) + with pytest.raises(DiffbotError, match="rate limit") as caught: + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x"})) + assert TOKEN not in str(caught.value) + + +# ------------------------------------------------------------ extract + + +def test_extract_normalizes_bare_domains(serve): + seen = serve(lambda r: httpx.Response(200, json={"objects": []})) + list(ExtractTool.from_credentials(CREDS).invoke({"url": "example.org/story"})) + assert seen[0].url.params["url"] == "https://example.org/story" + + +@pytest.mark.parametrize("url", ["localhost", "127.0.0.1/admin", "10.1.2.3", "metadata.google.internal"]) +def test_extract_rejects_private_bare_domains(url): + with pytest.raises(DiffbotError, match="not allowed"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": url})) + + +def test_extract_options_reach_the_request(serve): + seen = serve(lambda r: httpx.Response(200, json={"objects": []})) + params = { + "url": "https://example.org", + "fields": "links, meta", + "timeout": 60000, + "mode": "llm", + "paging": True, + "fallback": "article", + "use_proxy": "none", + "natural_language": "entities, summary", + "max_pages": 3, + "tag_confidence": 0.5, + } + list(ExtractTool.from_credentials(CREDS).invoke(params)) + q = seen[0].url.params + assert q["fields"] == "links,meta" + assert q["timeout"] == "60000" + assert q["mode"] == "llm" + assert q["paging"] == "true" + assert q["maxPages"] == "3" + assert q["tagConfidence"] == "0.5" + assert q["fallback"] == "article" and q["useProxy"] == "none" + assert q["naturalLanguage"] == "entities,summary" + + +def test_extract_option_validation(): + with pytest.raises(DiffbotError, match="fields"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "fields": "a&token=b"})) + with pytest.raises(DiffbotError, match="timeout"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "timeout": 5})) + + +def test_extract_rejects_unknown_option_values(): + with pytest.raises(DiffbotError, match="mode"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "mode": "bogus"})) + with pytest.raises(DiffbotError, match="natural_language"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "natural_language": "x"})) + + +def test_extract_passes_options_to_sdks_that_support_them(): + calls = [] + + class Fake: + token = TOKEN + + def extract(self, url, api="analyze", fmt="markdown", *, mode=None, fields=None, timeout=None): + calls.append((url, api, fmt, mode, fields, timeout)) + return {"objects": []} + + sdk.extract(Fake(), "https://x.org", "article", "json", {"mode": "llm", "fields": ["links"], "timeout": 5000}) # type: ignore[arg-type] + assert calls == [("https://x.org", "article", "json", "llm", ["links"], 5000)] + + +# ------------------------------------------------------------ KG / search limits + + +def test_kg_offset_and_large_size_are_sent(serve): + seen = serve(lambda r: httpx.Response(200, json={"hits": 0, "data": []})) + list(SearchOrganizationsTool.from_credentials(CREDS).invoke({"name": "Diffbot", "size": 100, "offset": 200})) + body = json.loads(seen[0].content) + assert body["size"] == 100 and body["from"] == 200 + + +def test_web_search_allows_50_results_and_rejects_more(serve): + seen = serve(lambda r: httpx.Response(200, json={"search_results": []})) + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x", "num_results": 50, "max_tokens": 32000})) + assert seen[0].url.params["size"] == "50" + with pytest.raises(DiffbotError): + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x", "num_results": 51})) + + +# ------------------------------------------------------------ crawl + + +def test_crawl_start_never_watches_and_returns_the_job_name(serve): + seen = serve(lambda r: httpx.Response(200, json={})) + payload = _json( + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j1", "hops": 1})) + ) + assert payload["job_name"] == "j1" + assert len(seen) == 1 # create only: no status poll, no data fetch + assert seen[0].url.params["seeds"] == "https://example.org" + assert seen[0].url.params["maxHops"] == "1" + + +@pytest.mark.parametrize("seeds", ["http://127.0.0.1", "https://ok.example.org localhost"]) +def test_crawl_start_guards_seeds(seeds): + with pytest.raises(DiffbotError, match="not allowed"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": seeds})) + + +def test_crawl_start_guards_api_url(): + with pytest.raises(DiffbotError, match="not allowed"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "api_url": "http://localhost/x"})) + + +def _jobs(parameters): + return list(CrawlJobsTool.from_credentials(CREDS).invoke(parameters)) + + +def test_crawl_jobs_list_and_status(serve): + job = {"name": "j1", "jobStatus": {"status": 9, "message": "done"}, "objectsFound": 4, "secretish": "x"} + serve(lambda r: httpx.Response(200, json={"jobs": [job]})) + assert _json(_jobs({"action": "list"}))["jobs"][0]["name"] == "j1" + status = _json(_jobs({"action": "status", "job_name": "j1"}))["job"] + assert status["objectsFound"] == 4 and "secretish" not in status + + +def test_crawl_jobs_status_for_missing_job(serve): + serve(lambda r: httpx.Response(200, json={"jobs": []})) + with pytest.raises(DiffbotError, match="not find"): + _jobs({"action": "status", "job_name": "nope"}) + + +def test_crawl_results_fall_back_to_the_data_endpoint_and_parse_csv(serve): + csv_text = 'Url,Crawl Status\nhttps://a.example.org,Success\nhttps://b.example.org,Success\n' + seen = serve(lambda r: httpx.Response(200, text=csv_text)) + payload = _json(_jobs({"action": "results", "job_name": "j1", "result_type": "urls", "limit": 1})) + assert seen[0].url.path == "/v3/crawl/data" + assert seen[0].url.params["type"] == "urls" + assert seen[0].url.params["num"] == "1" + assert payload["returned"] == 1 and payload["truncated"] is True + assert payload["results"][0]["Url"] == "https://a.example.org" + + +def test_crawl_results_use_native_crawl_download_when_available(): + class Fake: + token = TOKEN + + def crawl_download(self, name, *, format="json", type=None, num=None): + return httpx.Response(200, json=[{"title": "t"}]) + + assert sdk.crawl_download(Fake(), "j", num=5) == [{"title": "t"}] # type: ignore[arg-type] + + +def test_crawl_delete_is_not_retried(serve): + seen = serve(lambda r: httpx.Response(503, json={"message": "down"})) + with pytest.raises(DiffbotError): + _jobs({"action": "delete", "job_name": "j1"}) + assert len(seen) == 1 + assert seen[0].url.params["delete"] == "1" + + +def test_crawl_delete_is_labelled_destructive(): + spec = yaml.safe_load(open("diffbot/tools/crawl_jobs.yaml")) + assert "destructive" in spec["description"]["human"]["en_US"].lower() + action = next(p for p in spec["parameters"] if p["name"] == "action") + assert "destructive" in action["human_description"]["en_US"].lower() + + +def test_all_tool_yamls_are_registered_and_exist(): + provider = yaml.safe_load(open("diffbot/provider/diffbot.yaml")) + for path in provider["tools"]: + spec = yaml.safe_load(open(f"diffbot/{path}")) + assert spec["extra"]["python"]["source"].startswith("tools/") + assert {"tools/crawl_start.yaml", "tools/crawl_jobs.yaml"} <= set(provider["tools"]) From b63cc8e3c3690bc02d2893df5afb9bd2957d14c5 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:11:10 -1000 Subject: [PATCH 04/15] fix(crawl): cap job names at 30 characters and detect truncated results The crawler answers a name over 30 characters with HTTP 200 + errorCode 505 and creates nothing (diffbot 3.2.0 does not surface that), so crawl_start now rejects such names up front. crawl_jobs results asked for num=limit, so 'truncated' could never be true; request limit+1 and default to the yaml's 100 rows. Fix KG search llm_descriptions that said 1000 instead of 100. Co-Authored-By: Claude Opus 5.5 --- diffbot/tools/crawl_jobs.py | 6 ++++-- diffbot/tools/crawl_start.py | 10 ++++++++-- diffbot/tools/crawl_start.yaml | 4 ++-- diffbot/tools/search_news.yaml | 2 +- diffbot/tools/search_organizations.yaml | 2 +- diffbot/tools/search_people.yaml | 2 +- tests/test_sdk_wiring.py | 17 ++++++++++++++++- 7 files changed, 33 insertions(+), 10 deletions(-) diff --git a/diffbot/tools/crawl_jobs.py b/diffbot/tools/crawl_jobs.py index e5f241e..7f552fb 100644 --- a/diffbot/tools/crawl_jobs.py +++ b/diffbot/tools/crawl_jobs.py @@ -14,6 +14,7 @@ ACTIONS = {"list", "status", "results", "delete"} RESULT_LIMIT = 1000 +DEFAULT_RESULT_LIMIT = 100 # matches the yaml default JOB_FIELDS = ("name", "jobStatus", "pageCrawlAttempts", "pageCrawlSuccesses", "pageProcessAttempts", "pageProcessSuccesses", "objectsFound", "urlsFound", "maxToCrawl", "maxToProcess") @@ -54,14 +55,15 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag elif action == "results": limit = int( bounded_number( - tool_parameters.get("limit") or RESULT_LIMIT, + tool_parameters.get("limit") or DEFAULT_RESULT_LIMIT, label="limit", minimum=1, maximum=RESULT_LIMIT, integer=True, ) ) kind = optional_text(tool_parameters, "result_type").lower() if kind not in ("", "urls"): raise DiffbotError("result_type must be empty or urls.") - data = crawl_download(client, job_name, format="json", type=kind or None, num=limit) + # Ask for one extra row so "truncated" can be detected. + data = crawl_download(client, job_name, format="json", type=kind or None, num=limit + 1) rows = data if isinstance(data, list) else [data] kept = [{k: _cell(v) for k, v in row.items()} if isinstance(row, dict) else row for row in rows[:limit]] payload = { diff --git a/diffbot/tools/crawl_start.py b/diffbot/tools/crawl_start.py index cc6517c..961d07d 100644 --- a/diffbot/tools/crawl_start.py +++ b/diffbot/tools/crawl_start.py @@ -11,6 +11,8 @@ from tools.dql_builder import bounded_number from tools.sdk import crawl_start +MAX_JOB_NAME = 30 + def _int(parameters: dict[str, Any], name: str, *, minimum: int, maximum: int) -> int | None: value = parameters.get(name) @@ -37,8 +39,12 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag kwargs: dict[str, Any] = {} name = optional_text(tool_parameters, "job_name") if name: - if not all(c.isalnum() or c in "-_." for c in name) or len(name) > 100: - raise DiffbotError("job_name may use letters, digits, '-', '_' and '.' (100 characters maximum).") + # The crawler rejects names over 30 characters with HTTP 200 + errorCode 505 + # and creates nothing; diffbot 3.2.0 does not surface that, so check here. + if not all(c.isalnum() or c in "-_." for c in name) or len(name) > MAX_JOB_NAME: + raise DiffbotError( + f"job_name may use letters, digits, '-', '_' and '.' ({MAX_JOB_NAME} characters maximum)." + ) kwargs["job_name"] = name for param, key, low, high in ( ("max_to_crawl", "max_to_crawl", 1, 10000), diff --git a/diffbot/tools/crawl_start.yaml b/diffbot/tools/crawl_start.yaml index c3074ea..8a5728d 100644 --- a/diffbot/tools/crawl_start.yaml +++ b/diffbot/tools/crawl_start.yaml @@ -27,8 +27,8 @@ parameters: label: en_US: Job name human_description: - en_US: Optional name (letters, digits, - _ .). Defaults to crawl-. - llm_description: Optional job name. Leave empty to auto-generate. + en_US: Optional name (letters, digits, - _ .; 30 characters maximum). Defaults to crawl-. + llm_description: Optional job name, at most 30 characters (letters, digits, - _ .). Leave empty to auto-generate. - name: max_to_crawl type: number required: false diff --git a/diffbot/tools/search_news.yaml b/diffbot/tools/search_news.yaml index 9893f90..a204347 100644 --- a/diffbot/tools/search_news.yaml +++ b/diffbot/tools/search_news.yaml @@ -97,7 +97,7 @@ parameters: en_US: Maximum results human_description: en_US: How many articles to return, from 1 to 100. - llm_description: Result cap from 1 to 1000. Default 5. + llm_description: Result cap from 1 to 100. Default 5. - name: offset type: number required: false diff --git a/diffbot/tools/search_organizations.yaml b/diffbot/tools/search_organizations.yaml index 3b43400..291140b 100644 --- a/diffbot/tools/search_organizations.yaml +++ b/diffbot/tools/search_organizations.yaml @@ -128,7 +128,7 @@ parameters: en_US: Maximum results human_description: en_US: How many organizations to return, from 1 to 100. - llm_description: Result cap from 1 to 1000. Default 5. + llm_description: Result cap from 1 to 100. Default 5. - name: offset type: number required: false diff --git a/diffbot/tools/search_people.yaml b/diffbot/tools/search_people.yaml index 134ab83..195e4af 100644 --- a/diffbot/tools/search_people.yaml +++ b/diffbot/tools/search_people.yaml @@ -69,7 +69,7 @@ parameters: en_US: Maximum results human_description: en_US: How many people to return, from 1 to 100. - llm_description: Result cap from 1 to 1000. Default 5. + llm_description: Result cap from 1 to 100. Default 5. - name: offset type: number required: false diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index cd650b5..f815e66 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -236,6 +236,21 @@ def test_crawl_start_guards_seeds(seeds): list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": seeds})) +def test_crawl_start_rejects_job_names_over_30_characters(serve): + seen = serve(lambda r: httpx.Response(200, json={})) + with pytest.raises(DiffbotError, match="30 characters"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j" * 31})) + assert seen == [] + payload = _json(list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j" * 30}))) + assert payload["job_name"] == "j" * 30 + + +def test_crawl_results_not_truncated_when_rows_fit(serve): + serve(lambda r: httpx.Response(200, json=[{"title": "a"}, {"title": "b"}])) + payload = _json(_jobs({"action": "results", "job_name": "j1", "limit": 2})) + assert payload["returned"] == 2 and payload["truncated"] is False + + def test_crawl_start_guards_api_url(): with pytest.raises(DiffbotError, match="not allowed"): list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "api_url": "http://localhost/x"})) @@ -265,7 +280,7 @@ def test_crawl_results_fall_back_to_the_data_endpoint_and_parse_csv(serve): payload = _json(_jobs({"action": "results", "job_name": "j1", "result_type": "urls", "limit": 1})) assert seen[0].url.path == "/v3/crawl/data" assert seen[0].url.params["type"] == "urls" - assert seen[0].url.params["num"] == "1" + assert seen[0].url.params["num"] == "2" # one extra row detects truncation assert payload["returned"] == 1 and payload["truncated"] is True assert payload["results"][0]["Url"] == "https://a.example.org" From 2b70ee813a0337b5a3f10cb20857ddce3ce202da Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:12:03 -1000 Subject: [PATCH 05/15] fix(errors): surface crawl job errors, non-JSON bodies, and Enhance errorCode - JobError carries plain text, so APIError.message was empty and users saw 'Diffbot API error 500: request failed'; show the (redacted) reason. - The crawl_download fallback returned an HTTP 200 + errorCode body (e.g. an unknown job) as a result row; raise JobError like diffbot 3.3.0 does. - A non-JSON body escaped mapped_errors as a raw JSONDecodeError; map it. - The Enhance wrapper ignored HTTP 200 + errorCode (the old client checked it). Co-Authored-By: Claude Opus 5.5 --- diffbot/tools/sdk.py | 14 +++++++++++++- tests/test_sdk_wiring.py | 20 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py index 8b71335..5356dbc 100644 --- a/diffbot/tools/sdk.py +++ b/diffbot/tools/sdk.py @@ -31,6 +31,7 @@ Diffbot, DiffbotError as SdkError, ExtractionError, + JobError, NotFoundError, RateLimitError, ValidationError, @@ -113,6 +114,9 @@ def api_error_message(exc: BaseException, token: str) -> str: if exc.error_code in (401, 403): return INVALID_TOKEN return f"Diffbot could not extract the page ({exc.error_code}): {redact(exc.error, token)[:300]}" + if isinstance(exc, JobError): + # JobError carries plain text, so APIError.message is empty; show the body. + return f"Diffbot crawl job error: {redact(exc.body or 'job failed', token)[:300]}" if isinstance(exc, NotFoundError): return "Diffbot did not find that resource." if isinstance(exc, APIError): @@ -134,6 +138,8 @@ def mapped_errors(client: Diffbot) -> Iterator[None]: raise except ValidationError as exc: raise DiffbotError(redact(str(exc), client.token)) from None + except json.JSONDecodeError: + raise DiffbotError("Diffbot returned a non-JSON response.") from None except (SdkError, httpx.HTTPError) as exc: raise DiffbotError(api_error_message(exc, client.token)) from None @@ -266,9 +272,13 @@ def crawl_download( if not data.strip(): return [] try: - return json.loads(data) + parsed = json.loads(data) except ValueError: return list(csv.DictReader(io.StringIO(data))) + if isinstance(parsed, dict) and parsed.get("errorCode"): + # Same as diffbot 3.3.0: e.g. an unknown job is HTTP 200 + errorCode. + raise JobError(job_name, f"Crawl data download failed: {parsed.get('error') or 'unexpected response'}") + return parsed # ---------------------------------------------------------------- enhance (plugin-side only) @@ -280,6 +290,8 @@ def enhance(client: Diffbot, params: Mapping[str, Any]) -> dict[str, Any]: data = response.json() if not isinstance(data, dict): raise DiffbotError("Diffbot Enhance returned an unexpected response.") + if data.get("errorCode"): # some failures come back as HTTP 200 + errorCode + raise APIError(response.status_code, response.text) return data diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index f815e66..e6edc89 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -15,6 +15,7 @@ from tools.client import DiffbotError from tools.crawl_jobs import CrawlJobsTool from tools.crawl_start import CrawlStartTool +from tools.enhance_entity import EnhanceEntityTool from tools.extract import ExtractTool from tools.search_organizations import SearchOrganizationsTool from tools.search_web import SearchWebTool @@ -83,6 +84,19 @@ def test_validation_does_not_treat_server_errors_as_a_bad_token(serve): assert "Invalid or unauthorized" not in str(caught.value) +def test_non_json_body_is_a_safe_error(serve): + serve(lambda r: httpx.Response(200, text="oops")) + with pytest.raises(DiffbotError, match="non-JSON"): + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x"})) + + +def test_enhance_error_code_in_a_200_body_is_an_error(serve): + serve(lambda r: httpx.Response(200, json={"errorCode": 500, "error": f"boom {TOKEN}"})) + with pytest.raises(DiffbotError, match="boom") as caught: + list(EnhanceEntityTool.from_credentials(CREDS).invoke({"entity_type": "Organization", "name": "Diffbot"})) + assert TOKEN not in str(caught.value) + + @pytest.mark.parametrize("value", ["abc", "4", "301", "-1"]) def test_request_timeout_is_validated(value): with pytest.raises(DiffbotError, match="timeout"): @@ -295,6 +309,12 @@ def crawl_download(self, name, *, format="json", type=None, num=None): assert sdk.crawl_download(Fake(), "j", num=5) == [{"title": "t"}] # type: ignore[arg-type] +def test_crawl_download_job_error_shows_the_reason(serve): + serve(lambda r: httpx.Response(200, json={"errorCode": 500, "error": "No collection record"})) + with pytest.raises(DiffbotError, match="No collection record"): + _jobs({"action": "results", "job_name": "j1"}) + + def test_crawl_delete_is_not_retried(serve): seen = serve(lambda r: httpx.Response(503, json={"message": "down"})) with pytest.raises(DiffbotError): From 5ca52e614c6dfdfd2037cfbba96bff373921e1cc Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:12:46 -1000 Subject: [PATCH 06/15] refactor: drop unused local SDK response types; flatten extract boolean check Only CrawlJob and ExtractResponse are read by the plugin; the DqlResponse, WebSearchResponse, and NlpResponse fallbacks were dead code. Co-Authored-By: Claude Opus 5.5 --- diffbot/tools/extract.py | 6 ++---- diffbot/tools/sdk_types.py | 40 +++++++------------------------------- 2 files changed, 9 insertions(+), 37 deletions(-) diff --git a/diffbot/tools/extract.py b/diffbot/tools/extract.py index daf8ce4..a11ba70 100644 --- a/diffbot/tools/extract.py +++ b/diffbot/tools/extract.py @@ -61,10 +61,8 @@ def _extract_options(parameters: dict[str, Any]) -> dict[str, Any]: number = bounded_number(value, label=name, minimum=low, maximum=high, integer=integer) options[name] = int(number) if integer else float(number) for name in _BOOLEANS: - value = parameters.get(name) - if isinstance(value, bool): - if value: - options[name] = True # false is the API default, so only send true + if parameters.get(name) is True: + options[name] = True # false is the API default, so only send true natural = [part for part in optional_text(parameters, "natural_language").replace(" ", "").split(",") if part] if natural: bad = [part for part in natural if part not in NATURAL_LANGUAGE] diff --git a/diffbot/tools/sdk_types.py b/diffbot/tools/sdk_types.py index db3f5be..2287883 100644 --- a/diffbot/tools/sdk_types.py +++ b/diffbot/tools/sdk_types.py @@ -1,8 +1,8 @@ """Response shapes the plugin reads from the Diffbot SDK. -TODO(diffbot 3.3.0): the SDK will ship response TypedDicts (``AccountResponse`` and -friends). Until it does these local, ``total=False`` equivalents describe only the -fields this plugin touches. Swap them for ``from diffbot.types import ...`` then. +TODO(diffbot 3.3.0): the SDK ships response TypedDicts in ``diffbot.types``. Until +the pin requires it, these local ``total=False`` equivalents describe only the +fields this plugin touches. Delete the fallback once ``diffbot>=3.3.0`` is pinned. """ from __future__ import annotations @@ -10,14 +10,7 @@ from typing import Any, TypedDict try: # diffbot >= 3.3.0 ships these; prefer them. - from diffbot.types import ( # type: ignore[attr-defined] - CrawlJob, - CrawlJobStatus, - DqlResponse, - EntitiesResponse as NlpResponse, - ExtractResponse, - WebSearchResponse, - ) + from diffbot.types import CrawlJob, ExtractResponse # type: ignore[attr-defined] SDK_TYPES = True except ImportError: # diffbot 3.2.x: local equivalents below @@ -25,37 +18,18 @@ if not SDK_TYPES: - class ExtractResponse(TypedDict, total=False): + + class ExtractResponse(TypedDict, total=False): # type: ignore[no-redef] objects: list[dict[str, Any]] humanLanguage: str errorCode: int error: str - - class WebSearchResponse(TypedDict, total=False): - query: list[str] - timeMs: int - search_results: list[dict[str, Any]] - - - class DqlResponse(TypedDict, total=False): - hits: int - data: list[dict[str, Any]] - dql: str - - - class NlpResponse(TypedDict, total=False): - entities: list[dict[str, Any]] - sentiment: float - language: str - - class CrawlJobStatus(TypedDict, total=False): status: int message: str - - class CrawlJob(TypedDict, total=False): + class CrawlJob(TypedDict, total=False): # type: ignore[no-redef] name: str jobStatus: CrawlJobStatus pageCrawlSuccesses: int From d05bfd80454792b05b58ffc9649dc72e10d09bdb Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:13:07 -1000 Subject: [PATCH 07/15] test: detect the SDK by a real symbol so conftest skips work under python -m pytest Without the SDK installed, 'import diffbot' resolves to the plugin's own diffbot/ directory as a namespace package when the repo root is on sys.path, so the SDK-dependent modules were not skipped. Co-Authored-By: Claude Opus 5.5 --- tests/conftest.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/conftest.py b/tests/conftest.py index 3e15f71..dbec5fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -6,7 +6,10 @@ """ try: - import diffbot # noqa: F401 + # Import a real SDK name: with the repo root on sys.path (``python -m pytest``), + # a bare ``import diffbot`` succeeds on the plugin's own ``diffbot/`` directory + # as a namespace package even when the SDK is missing. + from diffbot import Diffbot # noqa: F401 except ImportError: collect_ignore = ["test_tools_http.py", "test_sdk_wiring.py"] print("diffbot SDK not installed: skipping SDK-dependent tests (see tests/conftest.py)") From bd5adb7491a6228e99839e0ccc7b1b4547cd77ac Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:14:17 -1000 Subject: [PATCH 08/15] fix(transport): honor HTTP(S)_PROXY/NO_PROXY behind the host guard httpx stops reading proxy environment variables once a transport is passed explicitly, so wrapping httpx.HTTPTransport() in GuardedTransport silently dropped the proxy support the old requests client had. Build per-scheme proxied transports from the environment and respect NO_PROXY. Test mocks accept transport kwargs so the suite also passes with a proxy set. Co-Authored-By: Claude Opus 5.5 --- diffbot/tools/sdk.py | 27 ++++++++++++++++++++++++--- tests/test_sdk_wiring.py | 20 +++++++++++++++++++- tests/test_tools_http.py | 2 +- 3 files changed, 44 insertions(+), 5 deletions(-) diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py index 5356dbc..19285c6 100644 --- a/diffbot/tools/sdk.py +++ b/diffbot/tools/sdk.py @@ -20,6 +20,7 @@ import inspect import io import json +import urllib.request from collections.abc import Iterator, Mapping from contextlib import contextmanager from typing import Any, cast @@ -54,21 +55,41 @@ class GuardedTransport(httpx.BaseTransport): """Wraps another transport and refuses private/loopback/metadata hosts and - non-http(s) schemes before any socket is opened.""" + non-http(s) schemes before any socket is opened. + + With no ``inner`` transport it honors ``HTTP(S)_PROXY`` / ``ALL_PROXY`` / + ``NO_PROXY`` like the old ``requests`` client did: httpx ignores the environment + once an explicit transport is passed, which would break proxied Dify deployments. + """ def __init__(self, inner: httpx.BaseTransport | None = None, *, block_private_hosts: bool = True) -> None: - self._inner = inner or httpx.HTTPTransport() self._block = block_private_hosts + self._proxied: dict[str, httpx.BaseTransport] = {} + if inner is None: + proxies = urllib.request.getproxies() + for scheme in ("http", "https"): + proxy = proxies.get(scheme) or proxies.get("all") + if proxy: + self._proxied[scheme] = httpx.HTTPTransport(proxy=proxy) + self._inner = inner or httpx.HTTPTransport() + + def _transport_for(self, url: httpx.URL) -> httpx.BaseTransport: + proxied = self._proxied.get(url.scheme) + if proxied is None or urllib.request.proxy_bypass(url.host): + return self._inner + return proxied def handle_request(self, request: httpx.Request) -> httpx.Response: if request.url.scheme not in ("http", "https"): raise DiffbotError("Request scheme is not allowed.") if self._block: check_public_host(request.url.host or "") - return self._inner.handle_request(request) + return self._transport_for(request.url).handle_request(request) def close(self) -> None: self._inner.close() + for transport in self._proxied.values(): + transport.close() def _as_bool(value: Any, default: bool) -> bool: diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index e6edc89..e03bffd 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -50,7 +50,7 @@ def wrapped(request): seen.append(request) return handler(request) - monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda: httpx.MockTransport(wrapped)) + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda **_: httpx.MockTransport(wrapped)) return seen return install @@ -128,6 +128,24 @@ def test_transport_blocks_private_hosts_before_any_socket(): assert inner_calls == [] +def test_default_transport_honors_proxy_environment(monkeypatch): + for name in ("HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "all_proxy", "no_proxy"): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("HTTPS_PROXY", "http://proxy.example.org:3128") + monkeypatch.setenv("NO_PROXY", "nl.diffbot.com") + guard = sdk.GuardedTransport() + assert guard._transport_for(httpx.URL("https://api.diffbot.com/v3/analyze")) is guard._proxied["https"] + assert guard._transport_for(httpx.URL("https://nl.diffbot.com/v1/")) is guard._inner + assert guard._transport_for(httpx.URL("http://api.diffbot.com/")) is guard._inner # no HTTP_PROXY + guard.close() + + +def test_explicit_inner_transport_ignores_proxy_environment(monkeypatch): + monkeypatch.setenv("HTTPS_PROXY", "http://proxy.example.org:3128") + inner = httpx.MockTransport(lambda r: httpx.Response(200)) + assert sdk.GuardedTransport(inner)._transport_for(httpx.URL("https://api.diffbot.com/")) is inner + + def test_guard_can_be_disabled_for_self_hosted_gateways(): inner = httpx.MockTransport(lambda r: httpx.Response(200, json={"objects": []})) client = Diffbot( diff --git a/tests/test_tools_http.py b/tests/test_tools_http.py index 28893d1..3df6cf2 100644 --- a/tests/test_tools_http.py +++ b/tests/test_tools_http.py @@ -51,7 +51,7 @@ def handler(request: httpx.Request) -> httpx.Response: return httpx.Response(result.status_code, text=result.payload, headers=result.headers) return httpx.Response(result.status_code, json=result.payload, headers=result.headers) - monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda: httpx.MockTransport(handler)) + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda **_: httpx.MockTransport(handler)) return holder return install From 7323b7bae31e5b06ee0e3d92fc819dac7b5176b1 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:30:57 -1000 Subject: [PATCH 09/15] Remove the enhance_entity tool to match the Diffbot Python SDK surface Drops the tool YAML/implementation, its provider entry, the plugin-side enhance() transport wrapper and trim_enhance() shaper, the related tests, and every mention in README, PRIVACY, and SUBMISSION docs. Co-Authored-By: Claude Sonnet 5.5 --- README.md | 1 - SUBMISSION.md | 21 +++---- diffbot/PRIVACY.md | 3 +- diffbot/README.md | 7 +-- diffbot/provider/diffbot.yaml | 1 - diffbot/tools/enhance_entity.py | 67 ----------------------- diffbot/tools/enhance_entity.yaml | 91 ------------------------------- diffbot/tools/sdk.py | 16 ------ diffbot/tools/shape.py | 11 ---- tests/test_sdk_wiring.py | 8 --- tests/test_tools_http.py | 38 ------------- 11 files changed, 13 insertions(+), 251 deletions(-) delete mode 100644 diffbot/tools/enhance_entity.py delete mode 100644 diffbot/tools/enhance_entity.yaml diff --git a/README.md b/README.md index f4615da..1783365 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,6 @@ The installable plugin is in [`diffbot/`](diffbot/). Its [README](diffbot/README | Search news | same DQL endpoint, newest publication date first via `sortBy:date` | | Start crawl | `GET https://api.diffbot.com/v3/crawl` (create only; returns a job name) | | Crawl jobs | list / status / results / **delete (destructive)** on `/v3/crawl` and `/v3/crawl/data` | -| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` (plugin-side wrapper; not in the SDK) | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Saving the credential checks `GET https://api.diffbot.com/v4/account?days=1`. Get a free token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). diff --git a/SUBMISSION.md b/SUBMISSION.md index 6f2be4a..84c7a39 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -114,7 +114,6 @@ New tool plugin for Diffbot. Seven tools: - Web search on llm.diffbot.com - Extract page on api.diffbot.com (auto or a specific page type) - Knowledge Graph search for organizations, people, and news -- Enhance entity for one organization or person - Resolve entities in pasted text to Knowledge Graph ids Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). Saving the credential calls the Account API. The plugin does not crawl, run raw DQL, or accept email, phone, or IP as inputs. @@ -125,7 +124,7 @@ Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). - [ ] Medium risk - [x] High risk -Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. People search, Enhance, and entity resolution handle personal data from public profiles and from text the user pastes (names, employers, locations, LinkedIn URLs). The submission rules say to pick the higher level when more than one could apply. Search-only peers have been accepted as low risk; this plugin does more than search, so it is marked high. +Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. People search and entity resolution handle personal data from public profiles and from text the user pastes (names, employers, locations, LinkedIn URLs). The submission rules say to pick the higher level when more than one could apply. Search-only peers have been accepted as low risk; this plugin does more than search, so it is marked high. ## Required checks @@ -141,9 +140,9 @@ Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. Pe ## Security and privacy notes -- Input constraints and security boundaries: outbound hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The user cannot pass a raw URL for the plugin to request, and cannot pass raw DQL. Extract rejects non-http(s) URLs, URLs with userinfo, and literal local, private, or link-local hosts. Timeouts are 15 to 45 seconds. Result counts are capped (10 search rows, 3 enhance matches, 20,000 characters of text). Errors are rewritten so the token is not returned. Email and phone fields are dropped from Knowledge Graph records before they reach the model. +- Input constraints and security boundaries: outbound hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The user cannot pass a raw URL for the plugin to request, and cannot pass raw DQL. Extract rejects non-http(s) URLs, URLs with userinfo, and literal local, private, or link-local hosts. Timeouts are 15 to 45 seconds. Result counts are capped (10 search rows, 20,000 characters of text). Errors are rewritten so the token is not returned. Email and phone fields are dropped from Knowledge Graph records before they reach the model. - Arbitrary URL fetching: Extract asks Diffbot to fetch one user-supplied public http(s) URL. The plugin itself only calls the four Diffbot hosts above. Scheme, userinfo, and literal private-host checks are described above. -- Personal data: People search, Enhance, and entity resolution process names, public profile fields, and location text. They do not accept email, phone, or IP input. Outputs drop email and phone if the API includes them. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could be returned; the privacy policy tells users not to send data about children. +- Personal data: People search and entity resolution process names, public profile fields, and location text. They do not accept email, phone, or IP input. Outputs drop email and phone if the API includes them. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could be returned; the privacy policy tells users not to send data about children. - Financial data: Organization search can return public revenue and headcount figures. It does not move money. - No command execution, code execution, shell, SQL against a customer database, SSH/SFTP, file operations, or browser automation. - Queries, URLs, and pasted text are sent to Diffbot over HTTPS. The plugin does not store them. See PRIVACY.md and https://www.diffbot.com/company/privacy. @@ -162,7 +161,7 @@ Validator: exit code 0, blocking failures 0. package_contents, package_secrets, Unit tests: `pytest` 31 passed (mocked HTTP). Plugin start-up: `validator/test-plugin-install.py` with `INSTALL_METHOD=serverless` succeeded. -Live smoke test against the Diffbot API on 10 November 2026, calling each tool class directly with a real token: credential check, web search, extract, organization search, people search, news search (newest first), enhance, and entity resolution all returned results. +Live smoke test against the Diffbot API on 10 November 2026, calling each tool class directly with a real token: credential check, web search, extract, organization search, people search, news search (newest first), and entity resolution all returned results. ## Reviewer notes @@ -219,11 +218,11 @@ A third-party write-up of Marketplace tool installs on 16 October 2026 put Tavil ## v1 tool choice -`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. The Enhance HTTP API is still documented at `https://kg.diffbot.com/kg/v3/enhance`. +`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. This plugin does not expose Enhance, which matches the Diffbot Python SDK surface. -v1 keeps the tools a Dify user already expects from Tavily or Firecrawl (search and extract) and adds the Knowledge Graph lookups that distinguish Diffbot (organizations, people, news, enhance, entity linking). It does not expose raw DQL, so a model cannot send an unbounded query. It does not start crawls, which are background jobs and a clearer "crawl arbitrary URLs" case. Places, deals, ontology browse, and hit-count probes can be a later version once v1 has been installed for real. +v1 keeps the tools a Dify user already expects from Tavily or Firecrawl (search and extract) and adds the Knowledge Graph lookups that distinguish Diffbot (organizations, people, news, entity linking). It does not expose raw DQL, so a model cannot send an unbounded query. It does not start crawls, which are background jobs and a clearer "crawl arbitrary URLs" case. Places, deals, ontology browse, and hit-count probes can be a later version once v1 has been installed for real. -Authentication matches the docs, with one exception. Extract, Account, DQL, Enhance, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` is documented with `Authorization: Bearer`, and the plugin does that so the token is not written into the query string for that host. +Authentication matches the docs, with one exception. Extract, Account, DQL, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` is documented with `Authorization: Bearer`, and the plugin does that so the token is not written into the query string for that host. ## Results for the current package @@ -262,7 +261,6 @@ Live smoke test, 10 November 2026, each tool class called directly with a real t [PASS] search_organizations (1.5s): text=Organization search returned 2 record(s) of 4. 1. Diffbot — Menlo Park, United States diffbot.com American machine learning and knowledge management company 2. ROS Mobile Robots — Feldkirch, Aus | json keys=['dql', 'hits', 'returned', 'results'] n=2 | first={"name": "Diffbot", "diffbotUri": "http://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ"} [PASS] search_people (2.4s): employer "Diffbot" + title "CEO" -> 1 public profile (Diffbot's founder and CEO); grouped employments.{...} query [PASS] search_news (2.1s): text=News search returned 3 article(s) of 3552, newest publication date first. 1. Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein (WinFuture.de — d2026-11-10T19:07) https://winfut | json keys=['dql', 'hits', 'returned', 'results'] n=3 | first={"title": "Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein", "date": "d2026-11-10T19:07", "pageUrl": "https://winfuture.de/news,162580.html"} -[PASS] enhance_entity (2.0s): text=Enhance returned 1 organization match(es). 1. Diffbot (score 0.943500057890868) diffbot.com http://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ | json keys=['hits', 'matches'] n=1 | first={"score": 0.943500057890868} [PASS] resolve_entities (1.8s): text=Resolved 4 entities from the supplied text. - Mike Tung [person] id=ESGMaGV9uP0SuTmfPTtNEoA - Menlo Park [location, administrative area, city] id=EQiAUr370NpW1gvx-V4fTrQ - Diffbot [organization] id=EY | json keys=['sentiment', 'entities', 'dql_filter'] n=4 ``` @@ -279,15 +277,14 @@ Blockers before a real PR: Product and API risks: -- Free tokens may not include Web Search, Knowledge Graph, Enhance, and Natural Language together. The Account API can succeed while a later call returns 401 or 403. The 10 November live test used one Diffbot token with all of them enabled; a free-plan token has not been tried. +- Free tokens may not include Web Search, Knowledge Graph, and Natural Language together. The Account API can succeed while a later call returns 401 or 403. The 10 November live test used one Diffbot token with all of them enabled; a free-plan token has not been tried. - Organization industry is an exact category string. The MCP server resolves free text through the ontology. v1 will return empty results for "software" when the category is "Software Companies". Either document that sharply or add ontology resolution in a later version. - Job-title abbreviations are not expanded (`CEO` versus `Chief Executive Officer`). The MCP server does expand them. The tool description tells the model to try both forms, but people search can still under-count until expansion exists. - Web Search might also accept `token` as a query parameter. This plugin uses only the documented bearer header. If a token type rejects the header, the tool fails closed. That is a live test, not a guess to code around. -- Enhance can spend more credits when `size` is above 1. The cap is 3. `refresh` and `search=true` are not exposed because they recrawl or do an extra web merge and are slow. Say that in support docs so customers do not expect them. Review and policy risks: -- High risk is the honest box. A reviewer can still ask for a narrower tool set. Dropping Extract would remove user-supplied URL fetch, but people search would still be personal data. Dropping people, Enhance, and entity resolution would look more like Tavily and might be argued as medium, because the remaining URL fetch is still "user-provided URLs" in the medium definition and "proxy or crawl arbitrary URLs" in the high definition. The written rule is to choose the higher label. Do not mark this low because Exa did. Exa's PR text said it does not fetch arbitrary URLs. This plugin does ask Diffbot to fetch them. +- High risk is the honest box. A reviewer can still ask for a narrower tool set. Dropping Extract would remove user-supplied URL fetch, but people search would still be personal data. Dropping people and entity resolution would look more like Tavily and might be argued as medium, because the remaining URL fetch is still "user-provided URLs" in the medium definition and "proxy or crawl arbitrary URLs" in the high definition. The written rule is to choose the higher label. Do not mark this low because Exa did. Exa's PR text said it does not fetch arbitrary URLs. This plugin does ask Diffbot to fetch them. - Diffbot's privacy policy says the service is not directed at anyone 18 or under, and gives parents a contact address for privacy requests. The Knowledge Graph can still return a public profile of a younger person, and this plugin cannot filter by age. Call that out in review rather than claiming the plugin is age-safe. - Dify Cloud customers will send workspace prompts to Diffbot. Whether that needs a DPA or a customer-facing subprocessors note is a Diffbot legal question. The plugin policy links Diffbot's policy and says the plugin itself stores nothing. It does not answer the Dify customer's contract question. - A merged PR might not produce a listing. Watch the Marketplace after merge, as Exa had to. diff --git a/diffbot/PRIVACY.md b/diffbot/PRIVACY.md index f2ebafd..9a98ee5 100644 --- a/diffbot/PRIVACY.md +++ b/diffbot/PRIVACY.md @@ -18,10 +18,9 @@ Every tool call leaves the Dify workspace and is processed by Diffbot Technologi | Search query, result count, snippet size | `llm.diffbot.com` Web Search | Run a web search. The token is sent as a bearer header. | | Page URL and page type | `api.diffbot.com` Extract | Fetch and structure one public page. Diffbot retrieves the URL. | | Names, homepages, categories, places, employee and revenue bounds, article phrases, publishers, dates | `kg.diffbot.com` DQL | Search the Knowledge Graph. | -| Person or organization name, public URL, location text, employer, title | `kg.diffbot.com` Enhance | Match a Knowledge Graph record. | | Plain text from the user, up to 20,000 characters, and an optional language code | `nl.diffbot.com` | Recognize and link named entities. | -The plugin does not offer parameters for email address, phone number, or IP address, including on Enhance, where the Diffbot API supports those fields. If a Knowledge Graph response still contains email or phone values, the plugin drops them before returning the tool result. It also drops article HTML and, unless the news tool is asked for an excerpt, article body text. +The plugin does not offer parameters for email address, phone number, or IP address. If a Knowledge Graph response still contains email or phone values, the plugin drops them before returning the tool result. It also drops article HTML and, unless the news tool is asked for an excerpt, article body text. People and organization results can still contain personal data that is already public on the web: names, job titles, employers, cities, countries, homepage URLs, and LinkedIn profile URLs. Diffbot describes people in its index as search subjects whose information was published on publicly available sites. News results can quote public articles. Entity resolution can identify people named in text the user pasted. diff --git a/diffbot/README.md b/diffbot/README.md index dd3dfca..549607c 100644 --- a/diffbot/README.md +++ b/diffbot/README.md @@ -2,7 +2,7 @@ Search the public web and the Diffbot Knowledge Graph from a Dify agent or workflow. -Diffbot structures pages and keeps a knowledge graph of organizations, people, and articles. This plugin exposes a small v1 of that API: web search, page extraction, organization and people lookup, news search, entity enhancement, and named-entity resolution. A free token from [app.diffbot.com/get-started](https://app.diffbot.com/get-started) is enough to authenticate. +Diffbot structures pages and keeps a knowledge graph of organizations, people, and articles. This plugin exposes a small v1 of that API: web search, page extraction, organization and people lookup, news search, and named-entity resolution. A free token from [app.diffbot.com/get-started](https://app.diffbot.com/get-started) is enough to authenticate. Published by Diffbot. Source: https://github.com/diffbot/dify-diffbot. Support: support@diffbot.com. @@ -17,7 +17,6 @@ Published by Diffbot. Source: https://github.com/diffbot/dify-diffbot. Support: | Search news | same DQL endpoint | Articles by mentioned entity, phrase, publisher, language, or date. | | Start crawl | `GET https://api.diffbot.com/v3/crawl` | Start a background crawl of public seed URLs. Returns a job name, never results. | | Crawl jobs | `/v3/crawl`, `/v3/crawl/data` | List jobs, check status, fetch results (up to 1000 rows, truncated), or delete a job. **Delete is destructive** and removes the job and its data. | -| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` | Best Knowledge Graph match for a person or organization from a name, URL, or employer. | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Names in a passage, linked to Knowledge Graph ids when Diffbot can match them. | Raw DQL, ontology lookup, Ask, places, and deals are not in this version. Industry filters must already be exact category names such as `Software Companies`. The plugin does not resolve free text onto the taxonomy. @@ -31,11 +30,11 @@ Raw DQL, ontology lookup, Ask, places, and deals are not in this version. Indust Dify stores the value as a secret. The plugin sends it only to the Diffbot hosts listed in `manifest.yaml`. Web Search uses `Authorization: Bearer`. The other calls use the `token` query parameter, which is how those APIs authenticate. Saving the credential calls `GET https://api.diffbot.com/v4/account` with `days=1` and does not show the account payload. -The token needs access to Extract, Web Search, Knowledge Graph, Enhance, and Natural Language. A free plan may not include every product. If a call returns an authorization error after the account check succeeds, confirm that product is enabled for the token. +The token needs access to Extract, Web Search, Knowledge Graph, and Natural Language. A free plan may not include every product. If a call returns an authorization error after the account check succeeds, confirm that product is enabled for the token. ## Use the tools -In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits: up to 50 web results, 100 Knowledge Graph rows per call (page with Offset), 3 enhance matches, and 20,000 characters of text for entity resolution. Extract has optional caller-set options (fields, timeout, mode, paging, natural language, and others). +In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits: up to 50 web results, 100 Knowledge Graph rows per call (page with Offset), and 20,000 characters of text for entity resolution. Extract has optional caller-set options (fields, timeout, mode, paging, natural language, and others). Extract accepts `http` and `https` URLs; a bare domain such as `example.com` gets `https://`. Localhost, link-local, and private literal addresses are rejected before the request is sent. diff --git a/diffbot/provider/diffbot.yaml b/diffbot/provider/diffbot.yaml index 9294c84..8bac71f 100644 --- a/diffbot/provider/diffbot.yaml +++ b/diffbot/provider/diffbot.yaml @@ -47,7 +47,6 @@ tools: - tools/search_news.yaml - tools/crawl_start.yaml - tools/crawl_jobs.yaml - - tools/enhance_entity.yaml - tools/resolve_entities.yaml extra: python: diff --git a/diffbot/tools/enhance_entity.py b/diffbot/tools/enhance_entity.py deleted file mode 100644 index 27e0765..0000000 --- a/diffbot/tools/enhance_entity.py +++ /dev/null @@ -1,67 +0,0 @@ -"""Enhance a person or organization from partial public identifiers.""" - -from collections.abc import Generator -from typing import Any - -from dify_plugin import Tool -from dify_plugin.entities.tool import ToolInvokeMessage - -from tools.client import DiffbotError, validate_public_http_url -from tools.common import make_client, mapped_errors, messages, optional_text, result_size -from tools.sdk import enhance -from tools.shape import summarize_lines, trim_enhance - - -class EnhanceEntityTool(Tool): - def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: - entity_type = optional_text(tool_parameters, "entity_type") - if entity_type not in {"Organization", "Person"}: - raise DiffbotError("entity_type must be Organization or Person.") - name = optional_text(tool_parameters, "name") - url = optional_text(tool_parameters, "url") - location = optional_text(tool_parameters, "location") - employer = optional_text(tool_parameters, "employer") - title = optional_text(tool_parameters, "title") - if entity_type == "Organization" and (employer or title): - raise DiffbotError("employer and title apply only when entity_type is Person.") - if not any((name, url, employer)): - raise DiffbotError("Provide a name, a url, or, for a person, an employer.") - if url: - url = validate_public_http_url(url) - - params: dict[str, Any] = { - "type": entity_type, - "size": result_size(tool_parameters, default=1, maximum=3), - } - if name: - params["name"] = name - if url: - params["url"] = url - if location: - params["location"] = location - if employer: - params["employer"] = employer - if title: - params["title"] = title - - # Enhance is not part of the Diffbot SDK; this goes through the SDK's transport - # (retries, timeouts, error mapping) via a plugin-side wrapper. - client = make_client(self.runtime.credentials) - with client, mapped_errors(client): - data = enhance(client, params) - payload = trim_enhance(data, entity_type) - lines = [] - for index, match in enumerate(payload["matches"], start=1): - entity = match.get("entity") or {} - score = match.get("score") - score_text = f" (score {score})" if score is not None else "" - lines.append(f"{index}. {entity.get('name') or 'Unnamed match'}{score_text}") - if entity.get("homepageUri"): - lines.append(f" {entity['homepageUri']}") - if entity.get("diffbotUri"): - lines.append(f" {entity['diffbotUri']}") - summary = summarize_lines( - f"Enhance returned {len(payload['matches'])} {entity_type.lower()} match(es).", - lines, - ) - yield from messages(self, summary, payload) diff --git a/diffbot/tools/enhance_entity.yaml b/diffbot/tools/enhance_entity.yaml deleted file mode 100644 index 2178493..0000000 --- a/diffbot/tools/enhance_entity.yaml +++ /dev/null @@ -1,91 +0,0 @@ -identity: - name: enhance_entity - author: diffbot - label: - en_US: Enhance entity -description: - human: - en_US: Match a person or organization from a name, homepage, or employer and return the Knowledge Graph record Diffbot considers the best fit. - llm: > - Call Diffbot Enhance to resolve one person or organization from partial public identifiers. - entity_type is Organization or Person. Provide name, url, or for a person an employer. - employer and title are valid only for Person. Email, phone, and IP are intentionally not accepted. - size is 1 to 3. The response includes the match score and a trimmed entity record. - This does not refresh or recrawl origins. -parameters: - - name: entity_type - type: select - required: true - form: llm - label: - en_US: Entity type - human_description: - en_US: Whether to match an organization or a person. - llm_description: Organization or Person. - options: - - value: Organization - label: - en_US: Organization - - value: Person - label: - en_US: Person - - name: name - type: string - required: false - form: llm - label: - en_US: Name - human_description: - en_US: Name of the organization or person. - llm_description: Entity name. More specific names match better. - - name: url - type: string - required: false - form: llm - label: - en_US: URL - human_description: - en_US: Public homepage or profile URL. Local and private hosts are rejected. - llm_description: Public http(s) homepage or profile URL. - - name: location - type: string - required: false - form: llm - label: - en_US: Location - human_description: - en_US: Optional city, region, or country that disambiguates the name. - llm_description: Free-text location used only to disambiguate the match. - - name: employer - type: string - required: false - form: llm - label: - en_US: Employer - human_description: - en_US: Employer name. Only for a person. - llm_description: Employer name. Valid only when entity_type is Person. - - name: title - type: string - required: false - form: llm - label: - en_US: Title - human_description: - en_US: Job title. Only for a person, and only together with a name, URL, or employer. - llm_description: Job title. Valid only when entity_type is Person. Not sufficient on its own. - - name: size - type: number - required: false - form: llm - default: 1 - min: 1 - max: 3 - label: - en_US: Maximum matches - human_description: - en_US: How many candidate matches to return, from 1 to 3. Defaults to 1. - llm_description: Match cap from 1 to 3. Default 1. Each extra match can spend more credits. -extra: - python: - source: tools/enhance_entity.py diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py index 19285c6..2f9b48c 100644 --- a/diffbot/tools/sdk.py +++ b/diffbot/tools/sdk.py @@ -8,7 +8,6 @@ * ``verify_token`` -> diffbot 3.3.0 (fallback: ``GET /v4/account?days=1``) * ``crawl_download`` -> diffbot 3.3.0 (fallback: ``GET /v3/crawl/data``) * extract options -> diffbot 3.3.0 (fallback: ``GET /v3/{api}`` with the same params) -* Enhance -> never in the SDK; a thin plugin-side wrapper stays. TODO: when diffbot 3.2.0 / 3.3.0 are released, pin ``diffbot`` in ``requirements.txt`` / ``pyproject.toml`` and delete the fallbacks. @@ -302,20 +301,6 @@ def crawl_download( return parsed -# ---------------------------------------------------------------- enhance (plugin-side only) - - -def enhance(client: Diffbot, params: Mapping[str, Any]) -> dict[str, Any]: - """Thin wrapper over the public transport; the SDK deliberately has no Enhance.""" - response = client.request("GET", f"{client.kg_base}/kg/v3/enhance", params={"token": client.token, **params}) - data = response.json() - if not isinstance(data, dict): - raise DiffbotError("Diffbot Enhance returned an unexpected response.") - if data.get("errorCode"): # some failures come back as HTTP 200 + errorCode - raise APIError(response.status_code, response.text) - return data - - def as_dict(value: Any, what: str) -> dict[str, Any]: if not isinstance(value, dict): raise DiffbotError(f"Diffbot {what} returned an unexpected response.") @@ -331,5 +316,4 @@ def as_dict(value: Any, what: str) -> dict[str, Any]: "crawl_start", "crawl_get_job", "crawl_download", - "enhance", ] diff --git a/diffbot/tools/shape.py b/diffbot/tools/shape.py index c6384f1..3bfe45f 100644 --- a/diffbot/tools/shape.py +++ b/diffbot/tools/shape.py @@ -135,17 +135,6 @@ def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False, limi } -def trim_enhance(data: dict[str, Any], kind: str) -> dict[str, Any]: - matches = [] - for item in (data.get("data") or [])[:5]: - if not isinstance(item, dict): - continue - entity = item.get("entity") if isinstance(item.get("entity"), dict) else {} - trimmed = trim_organization(entity) if kind == "Organization" else trim_person(entity) - matches.append({"score": item.get("score"), "entity": trimmed}) - return {"hits": data.get("hits"), "matches": matches} - - def trim_extract(data: dict[str, Any]) -> dict[str, Any]: objects = data.get("objects") trimmed = [] diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index e03bffd..a7f8609 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -15,7 +15,6 @@ from tools.client import DiffbotError from tools.crawl_jobs import CrawlJobsTool from tools.crawl_start import CrawlStartTool -from tools.enhance_entity import EnhanceEntityTool from tools.extract import ExtractTool from tools.search_organizations import SearchOrganizationsTool from tools.search_web import SearchWebTool @@ -90,13 +89,6 @@ def test_non_json_body_is_a_safe_error(serve): list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x"})) -def test_enhance_error_code_in_a_200_body_is_an_error(serve): - serve(lambda r: httpx.Response(200, json={"errorCode": 500, "error": f"boom {TOKEN}"})) - with pytest.raises(DiffbotError, match="boom") as caught: - list(EnhanceEntityTool.from_credentials(CREDS).invoke({"entity_type": "Organization", "name": "Diffbot"})) - assert TOKEN not in str(caught.value) - - @pytest.mark.parametrize("value", ["abc", "4", "301", "-1"]) def test_request_timeout_is_validated(value): with pytest.raises(DiffbotError, match="timeout"): diff --git a/tests/test_tools_http.py b/tests/test_tools_http.py index 3df6cf2..8b3ec08 100644 --- a/tests/test_tools_http.py +++ b/tests/test_tools_http.py @@ -7,7 +7,6 @@ from provider.diffbot import DiffbotProvider from tools.client import TOKEN_PATTERN, DiffbotError -from tools.enhance_entity import EnhanceEntityTool from tools.extract import ExtractTool from tools.resolve_entities import ResolveEntitiesTool from tools.search_news import SearchNewsTool @@ -271,43 +270,6 @@ def responder(method, url, kwargs): assert "text" not in payload["results"][0] -def test_enhance_sends_partial_identifiers(session_factory): - def responder(method, url, kwargs): - assert kwargs["params"]["type"] == "Organization" - assert kwargs["params"]["name"] == "Diffbot" - assert "email" not in kwargs["params"] - return FakeResponse( - 200, - { - "hits": 1, - "data": [ - { - "score": 0.7, - "entity": { - "name": "Diffbot", - "homepageUri": "https://www.diffbot.com", - "diffbotUri": "http://diffbot.com/entity/org1", - "allEmail": ["secret@example.com"], - }, - } - ], - }, - ) - - session_factory(responder) - tool = EnhanceEntityTool.from_credentials({"diffbot_token": TOKEN}) - payload = _json_message(list(tool.invoke({"entity_type": "Organization", "name": "Diffbot"}))) - assert payload["matches"][0]["score"] == 0.7 - assert payload["matches"][0]["entity"]["id"] == "org1" - assert "secret@example.com" not in json.dumps(payload) - - -def test_enhance_rejects_employer_for_an_organization(): - tool = EnhanceEntityTool.from_credentials({"diffbot_token": TOKEN}) - with pytest.raises(DiffbotError, match="employer"): - list(tool.invoke({"entity_type": "Organization", "name": "Diffbot", "employer": "Nobody"})) - - def test_resolve_entities_builds_a_dql_filter(session_factory): def responder(method, url, kwargs): assert method == "POST" From c298147ad756a23b54d4b9bb266ce84b732cc077 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:53:23 -1000 Subject: [PATCH 10/15] Rewrite SUBMISSION.md for the SDK-based plugin with crawl Describe the eight current tools (enhance_entity removed in 7323b7b), crawl_start/crawl_jobs behavior (background jobs, 30-character job names, destructive delete, account-wide job scope), the diffbot SDK + httpx transport replacing requests, the three credential fields, the four network domains, and a crawl-aware high-risk narrative. Drop the pre-SDK validator numbers. The package cannot be revalidated until diffbot 3.4.0 is on PyPI, so those results are marked TODO(sdk-release): rerun marketplace validator. Co-Authored-By: Claude Opus 5.5 --- SUBMISSION.md | 200 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 130 insertions(+), 70 deletions(-) diff --git a/SUBMISSION.md b/SUBMISSION.md index 84c7a39..4b5feb9 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,7 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. As of 10 November 2026 the source is public at https://github.com/diffbot/dify-diffbot and the package passes the marketplace validator, but no pull request has been opened against `langgenius/dify-plugins` and the package has not been installed in Dify Community Edition or Dify Cloud. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0` at release) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 is not on PyPI, so the package cannot be built with its real dependency pin. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. + Checked on 9 November 2026 against: @@ -15,6 +16,7 @@ Checked on 9 November 2026 against: - Marketplace search API `POST https://marketplace.dify.ai/api/v1/plugins/search/advanced` and `POST https://marketplace.dify.ai/api/v1/plugins/batch` - Diffbot docs at docs.diffbot.com and the public tool list in [diffbot/diffbot-mcp](https://github.com/diffbot/diffbot-mcp) + ## Status before a PR Done: @@ -24,17 +26,80 @@ Done: - Contact: `support@diffbot.com` in `manifest.yaml`, the plugin README, PRIVACY.md, and the PR body. - Icon: `_assets/icon.png` is the official 2026 Diffbot logomark, the green rounded-square tile (512×512 PNG). - `meta.minimum_dify_version`: 1.14.0. The dify_plugin 0.11.0 README lists Dify 1.14.0 for SDK 0.8.1 and later. The plugin uses no later gated feature (OAuth, dynamic-select, multiple selects, reset_on_change). -- Live API check: every tool was called against the real Diffbot API on 10 November 2026. Results are below. +- Unit tests: `pytest` reports 109 passed on 17 November 2026, with mocked HTTP, run against a local editable checkout of the unreleased SDK (`requirements-dev.txt`). Still to do: -1. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. -2. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. -3. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. +1. **Release `diffbot` 3.4.0 to PyPI and pin it.** `diffbot/requirements.txt` and `diffbot/pyproject.toml` still carry a `TODO(release)` comment in place of the pin, and that comment names 3.2.0/3.3.0. Replace it with `diffbot==3.4.0` in both files. Delete the `-e /workspace/diffbot-python` line from `requirements-dev.txt`. Then delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options). +2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. +3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. +4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. +5. **Update `diffbot/PRIVACY.md` for crawl.** Its "What is sent to Diffbot" table has no rows for seed URLs, crawl patterns, custom headers, or crawl results, and it does not say that crawl jobs and their data stay in the user's Diffbot account until someone deletes them. The privacy text has to match the package before submission. +6. Decide what to do with the `block_private_hosts` credential (see "Open questions and risks"). +7. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. +8. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, crawl seeds, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. + +## What the plugin is now + +### Tools + +`provider/diffbot.yaml` registers eight tools, one YAML and one Python file each under `diffbot/tools/`: + +| Tool | Diffbot host | What it does | +| --- | --- | --- | +| `search_web` | llm.diffbot.com | Web search. 1 to 50 results (default 5), optional snippet length and token budget. A query that is only `url:` returns the indexed copy of that page. | +| `extract` | api.diffbot.com | Diffbot fetches one public page and returns structured content (auto/analyze, article, product, discussion, image, video, list, event), as markdown or JSON, with optional Extract API options. | +| `search_organizations` | kg.diffbot.com | Knowledge Graph DQL built from typed fields: name, homepage, industry category, location, headcount and revenue bounds, public-company status. Up to 100 records. | +| `search_people` | kg.diffbot.com | Knowledge Graph search for people with a public professional presence, by name, employer, title, location. Up to 100 records. | +| `search_news` | kg.diffbot.com | Knowledge Graph article search by mentioned entity, phrase, publisher, language, and date, newest first. Up to 100 records. | +| `crawl_start` | api.diffbot.com (`/v3/crawl`) | Starts a background Diffbot crawl job on 1 to 20 public seed URLs and returns only the job name. | +| `crawl_jobs` | api.diffbot.com (`/v3/crawl`, `/v3/crawl/data`) | `list`, `status`, `results` (up to 1,000 rows, flagged when truncated), or `delete`. Delete is destructive. | +| `resolve_entities` | nl.diffbot.com | Finds entities in up to 20,000 characters of pasted text and links them to Knowledge Graph ids, with sentiment. | + +The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches the Diffbot Python SDK surface. The plugin has no raw DQL tool. + +### Crawl support + +- **Background jobs.** `crawl_start` always calls the SDK with `watch=False` and returns as soon as Diffbot reports the job name. Polling would run past the plugin's 300-second call limit. The job keeps running on Diffbot after the Dify call returns, and it uses the account's crawl quota until it finishes or someone deletes it. The model is told to check `crawl_jobs` `status` and then fetch `results`. +- **Job names, 30 characters max.** Optional `job_name` may use letters, digits, `-`, `_`, and `.`, up to 30 characters. Leave it empty and Diffbot generates `crawl-`. The plugin checks the length itself because the crawler answers a longer name with HTTP 200 plus `errorCode` 505 and creates nothing. +- **Bounds.** Up to 20 seeds. `max_to_crawl` and `max_to_process` are 1 to 10,000 (default 100). `hops` is 0 to 10 (default 2). `restrict_domain` is on by default. `crawl_delay` is −1 to 3,600 seconds (−1 means the Diffbot default). Optional URL crawl and process patterns, `obey_robots` (form default off; see the robots.txt note under risks), `use_proxies` (default off), a custom request header, and an optional `api_url` for per-page extraction. +- **Delete is destructive.** `crawl_jobs` `delete` permanently removes the job and the data it collected. There is no undo. The tool, action label, and LLM description all say so, and the LLM description tells the model to delete only when the user explicitly asks. The plugin itself has no confirmation step, so an agent that ignores that instruction can still delete a job. +- **Account scope.** `list`, `status`, `results`, and `delete` work on every crawl job owned by the token's Diffbot account, including jobs made outside Dify. Nothing limits a Dify workspace to the jobs it started. +- **Results.** Up to 1,000 rows, string cells clipped to 500 characters. The plugin asks for one extra row so it can set `truncated`. `result_type: urls` returns the crawled-URL report (CSV, parsed to rows). + +### Dependencies and transport + +- `dify_plugin==0.11.0`. +- `diffbot==3.4.0`, the official Diffbot Python SDK. **Not pinned yet**: it goes into `requirements.txt` and `pyproject.toml` once 3.4.0 is on PyPI. Until then the tests run against a local editable checkout. +- `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. +- Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and, while `block_private_hosts` is on, loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. +- Calls the SDK does not have yet go through the SDK's public `client.request()`, never its private attributes: token verification (`GET /v4/account?days=1`), crawl data download, and extract options. Each one feature-detects the native SDK method and uses it when present. +- The SDK sends the token as a `token` query parameter to api, kg, and nl.diffbot.com, and as `Authorization: Bearer` to llm.diffbot.com (Web Search). Errors are mapped to plugin-safe text and the token is redacted from them. + +### Credential fields + +From `provider/diffbot.yaml` `credentials_for_provider`: + +| Field | Type | Required | Default | Purpose | +| --- | --- | --- | --- | --- | +| `diffbot_token` | secret-input | yes | none | Diffbot API token. Format-checked (16 to 256 characters of `A-Za-z0-9._~+-`) and verified with the Account API when the credential is saved. | +| `request_timeout` | text-input | no | `30` | Seconds per Diffbot call, 5 to 300. The 300 ceiling is dify_plugin's request limit. | +| `block_private_hosts` | boolean | no | `true` | Toggles the transport host guard described above. See the open question about this field below. | + +### Network domains + +`manifest.yaml` `network.domains`, unchanged from v0.1: + +- `api.diffbot.com` (Extract, Account, Crawl) +- `kg.diffbot.com` (Knowledge Graph DQL) +- `llm.diffbot.com` (Web Search) +- `nl.diffbot.com` (Natural Language) + +The plugin process only talks to these four hosts. Neither credentials nor parameters can change the SDK base URLs. User-supplied URLs (extract `url`, crawl `seeds`, crawl `api_url`) go to Diffbot as parameters, and Diffbot's own infrastructure fetches them. The plugin does not. ## Package the file they will attach -From a checkout of the plugin, with the Dify plugin CLI (this run used `dify-plugin-linux-amd64` 0.6.15): +From a checkout of the plugin, after `diffbot==3.4.0` is pinned, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): ```bash dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg @@ -42,7 +107,7 @@ dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report ``` -The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below already contains the summary for the current package. +The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below has `TODO(sdk-release)` placeholders where the new run's summary goes. `dify plugin package` produces an unsigned archive. Dify signs marketplace packages after merge. A third-party signature is only for Community Edition installs outside the marketplace: @@ -89,7 +154,7 @@ Later versions add a new file such as `diffbot/diffbot/diffbot-0.2.0.difypkg` an ## PR body to paste -Fill this from the live template. Do not delete sections. The risk box must be exactly one level. None or several becomes the `risk: missing` label. +Fill this in from the live template. Keep every section. Check exactly one risk box: none, or more than one, gets the `risk: missing` label. Do not paste it as-is. Every `TODO(sdk-release)` line has to be replaced with real results first. ````markdown # Plugin Submission @@ -109,14 +174,15 @@ Fill this from the live template. Do not delete sections. The risk box must be e ## What changed -New tool plugin for Diffbot. Seven tools: +New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0`). Eight tools: - Web search on llm.diffbot.com -- Extract page on api.diffbot.com (auto or a specific page type) +- Extract a page on api.diffbot.com (auto or a specific page type) - Knowledge Graph search for organizations, people, and news +- Start a Diffbot crawl job, and list, check, fetch results of, or delete crawl jobs (delete is destructive) - Resolve entities in pasted text to Knowledge Graph ids -Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). Saving the credential calls the Account API. The plugin does not crawl, run raw DQL, or accept email, phone, or IP as inputs. +Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). The plugin calls the Account API to check the token when the credential is saved. Two optional credential fields set the request timeout (5 to 300 seconds) and a private-host guard. The plugin has no raw DQL tool and takes no email, phone, or IP inputs. ## Risk level @@ -124,7 +190,7 @@ Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). - [ ] Medium risk - [x] High risk -Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. People search and entity resolution handle personal data from public profiles and from text the user pastes (names, employers, locations, LinkedIn URLs). The submission rules say to pick the higher level when more than one could apply. Search-only peers have been accepted as low risk; this plugin does more than search, so it is marked high. +Extract and crawl both send user-supplied http(s) URLs to Diffbot, and Diffbot fetches them. A crawl follows links from up to 20 seeds, up to 10,000 pages, and keeps running in the background on the user's Diffbot account after the tool call returns. That falls under "proxy or crawl arbitrary URLs" in the high-risk definition. The crawl jobs tool can permanently delete a crawl job and its data. People search and entity resolution handle personal data from public profiles and from text the user pastes. ## Required checks @@ -138,41 +204,50 @@ Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. Pe - [x] The plugin includes `PRIVACY.md` or a hosted privacy policy, and `manifest.yaml` references it. - [x] All user-facing text is primarily in English, with any localized README files following the [i18n guidance](https://docs.dify.ai/en/develop-plugin/features-and-specs/plugin-types/multilingual-readme). + + ## Security and privacy notes -- Input constraints and security boundaries: outbound hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The user cannot pass a raw URL for the plugin to request, and cannot pass raw DQL. Extract rejects non-http(s) URLs, URLs with userinfo, and literal local, private, or link-local hosts. Timeouts are 15 to 45 seconds. Result counts are capped (10 search rows, 20,000 characters of text). Errors are rewritten so the token is not returned. Email and phone fields are dropped from Knowledge Graph records before they reach the model. -- Arbitrary URL fetching: Extract asks Diffbot to fetch one user-supplied public http(s) URL. The plugin itself only calls the four Diffbot hosts above. Scheme, userinfo, and literal private-host checks are described above. -- Personal data: People search and entity resolution process names, public profile fields, and location text. They do not accept email, phone, or IP input. Outputs drop email and phone if the API includes them. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could be returned; the privacy policy tells users not to send data about children. -- Financial data: Organization search can return public revenue and headcount figures. It does not move money. +- Outbound hosts: the plugin process only calls api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com, through the Diffbot SDK over httpx. No credential or parameter changes those base URLs. A guarded httpx transport refuses non-http(s) schemes and private, loopback, link-local, and metadata hosts before connecting. +- User-supplied URLs: extract `url`, crawl `seeds`, and crawl `api_url` must be http(s), have no userinfo, be at most 2,048 characters, and not be a local name or a literal private, loopback, link-local, reserved, or metadata address. Bare domains get `https://`. The check is on the literal host only. The plugin does not resolve DNS, because Diffbot does the fetching from its own infrastructure. +- Crawling: crawl jobs run on Diffbot in the background and use the account's quota until they finish or are deleted. Limits: 20 seeds, 10,000 pages crawled and processed, 10 hops. The crawl stays on the seed domains by default. `obey_robots` is off in the form by default. When it is off the SDK leaves out `obeyRobots`, so the Diffbot crawler's own default applies. A custom request header can be set and is sent to the crawled sites, and the parameter text tells users not to put secrets there. Job names are at most 30 characters. +- Destructive action: `crawl_jobs` with `delete` permanently removes a crawl job and its collected data. The action is labeled destructive, and the model is told to delete only on an explicit user request. List, status, results, and delete cover every crawl job on the token's account, not only jobs started from Dify. +- Limits: each call has a configurable timeout of 5 to 300 seconds (default 30). Web search returns at most 50 results, and Knowledge Graph searches return at most 100 records. Crawl results return at most 1,000 rows with cells clipped to 500 characters. Entity resolution accepts up to 20,000 characters. Long text fields are clipped before they reach the model. Errors are rewritten so they never contain the token. Email and phone fields are removed from Knowledge Graph records. +- Personal data: people search and entity resolution process names, public profile fields, and location text. Neither tool accepts email, phone, or IP input. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could come back. The privacy policy tells users not to send data about children. +- Financial data: organization search can return public revenue and headcount figures. The plugin does not move money. - No command execution, code execution, shell, SQL against a customer database, SSH/SFTP, file operations, or browser automation. -- Queries, URLs, and pasted text are sent to Diffbot over HTTPS. The plugin does not store them. See PRIVACY.md and https://www.diffbot.com/company/privacy. +- Queries, URLs, crawl settings, and pasted text go to Diffbot over HTTPS. The plugin stores none of it. Crawl jobs and their results are kept in the user's Diffbot account until deleted. See PRIVACY.md and https://www.diffbot.com/company/privacy. ## Local validation ```text -dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI 0.6.15 +dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report --pr-body-file pr-body.md ``` -Checksum `6abed6eb542446e656d598ed5bb98476cc2ec971a3254a7f295661ce10ca84a8`. +Checksum: TODO(sdk-release): rerun marketplace validator. -Validator: exit code 0, blocking failures 0. package_contents, package_secrets, package_binaries, manifest_metadata, readme_metadata, package_dependencies, python_compile, sensitive_capabilities: PASS. python_safety, prohibited_financial_activity: PASS. access_domains: static scan found only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, nl.diffbot.com, matching `network.domains`. dependency_vulnerabilities: `dify_plugin==0.11.0` and `requests==2.34.2` pinned, no known vulnerabilities. +Validator: TODO(sdk-release): rerun marketplace validator. Report exit code, blocking failures, and each check, especially: +- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0`. Record what OSV reports, including transitive `httpx`. +- access_domains: expected hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The base URLs now sit inside the SDK, so record what the static scan actually finds. +- python_safety: record the result for the new SDK/httpx code (`tools/sdk.py`, crawl tools). -Unit tests: `pytest` 31 passed (mocked HTTP). Plugin start-up: `validator/test-plugin-install.py` with `INSTALL_METHOD=serverless` succeeded. +Unit tests: `pytest` 109 passed (mocked HTTP). Plugin start-up (`validator/test-plugin-install.py`, `INSTALL_METHOD=serverless`): TODO(sdk-release): rerun marketplace validator. -Live smoke test against the Diffbot API on 10 November 2026, calling each tool class directly with a real token: credential check, web search, extract, organization search, people search, news search (newest first), and entity resolution all returned results. +Live smoke test: TODO(sdk-release): rerun every tool, including a crawl start/status/results/delete cycle, against the Diffbot API with a real token. ## Reviewer notes -- Not yet installed on Dify Community Edition or Dify Cloud. The tools were exercised live outside Dify, and the package passes the toolkit's start-up test. Results from a Community Edition and Dify Cloud install will be added to this PR before it is marked ready for review. -- `meta.minimum_dify_version` is 1.14.0, from the dify_plugin 0.11.0 compatibility table (SDK 0.8.1 and later). The plugin uses no later gated feature. +- Not yet installed on Dify Community Edition or Dify Cloud. Results from both will be added to this PR before it is marked ready for review. +- `meta.minimum_dify_version` is 1.14.0, taken from the dify_plugin 0.11.0 compatibility table (SDK 0.8.1 and later). The plugin uses no later gated feature. +- Crawl is asynchronous on purpose. A Dify tool call is capped at 300 seconds, so `crawl_start` returns a job name and the agent polls with `crawl_jobs`. - Industry filter on organization search is an exact `categories.name`. Free text is not resolved through the ontology in this version. -- Job titles match profile text as written. `CEO` and `Chief Executive Officer` may return different people; the tool description tells the model to try both. +- Job titles match profile text as written. `CEO` and `Chief Executive Officer` may return different people. The tool description tells the model to try both. - The icon is Diffbot's own 2026 logomark, published by Diffbot. ```` -Leave the "tested on Community Edition and Dify Cloud" box unchecked until that test exists. Do not check a box the package does not meet. Explain the gap under reviewer notes instead. +Leave the "tested on Community Edition and Dify Cloud" box unchecked until that test has been done. Never check a box the package does not meet. Explain the gap under reviewer notes instead. ## What reviewers look for @@ -216,77 +291,62 @@ No Diffbot Dify plugin showed up in the Marketplace search, in the Diffbot GitHu A third-party write-up of Marketplace tool installs on 16 October 2026 put Tavily, Jina, and Firecrawl in the same band (about 399k, 227k, and 204k). The 9 November API numbers above are higher and are the ones to cite. + ## v1 tool choice -`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. This plugin does not expose Enhance, which matches the Diffbot Python SDK surface. +`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. This plugin dropped its `enhance_entity` tool for the same reason (commit 7323b7b), so it matches the Diffbot Python SDK surface. -v1 keeps the tools a Dify user already expects from Tavily or Firecrawl (search and extract) and adds the Knowledge Graph lookups that distinguish Diffbot (organizations, people, news, entity linking). It does not expose raw DQL, so a model cannot send an unbounded query. It does not start crawls, which are background jobs and a clearer "crawl arbitrary URLs" case. Places, deals, ontology browse, and hit-count probes can be a later version once v1 has been installed for real. +v1 has what a Dify user expects from Tavily or Firecrawl: search, extract, and crawl. It adds the Knowledge Graph lookups that set Diffbot apart: organizations, people, news, and entity linking. There is no raw DQL tool, so a model cannot send an unbounded query. Crawl is in v1 because Tavily and Firecrawl both offer it. That decision moves the plugin into the "crawl arbitrary URLs" high-risk case. Places, deals, ontology browse, and hit-count probes can come in a later version once v1 has been installed for real. -Authentication matches the docs, with one exception. Extract, Account, DQL, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` is documented with `Authorization: Bearer`, and the plugin does that so the token is not written into the query string for that host. +The SDK handles authentication as the API docs describe. Extract, Account, Crawl, DQL, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` uses `Authorization: Bearer`, so the token does not appear in that host's query string. ## Results for the current package -Unit tests, no network: +Unit tests, no network, 17 November 2026, against a local editable checkout of the unreleased SDK: ```text pytest -31 passed +109 passed ``` -Dify CLI 0.6.15: - -```text -dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg -dify plugin checksum dist/diffbot-0.1.0.difypkg -# checksum 6abed6eb542446e656d598ed5bb98476cc2ec971a3254a7f295661ce10ca84a8 -``` - -The archive is about 62 KB, 29 files: the manifest, provider, seven tool pairs, client code, README, PRIVACY.md, icon, `main.py`, `requirements.txt`, and `pyproject.toml`. `.env.example` is kept in git for remote debugging and excluded from the package by `.difyignore`. `dist/` is not tracked in git. - -Marketplace toolkit validator, with `--pr-body-file` and the OSV lookup: +Package, checksum, archive size and file list: **TODO(sdk-release): rerun marketplace validator.** The package should hold the manifest, provider, eight tool YAML/Python pairs, the shared modules (`client.py`, `common.py`, `sdk.py`, `sdk_types.py`, `dql_builder.py`, `shape.py`), README, PRIVACY.md, icon, `main.py`, `requirements.txt`, and `pyproject.toml`. Check the real archive against that list. `.env.example` stays in git for remote debugging and `.difyignore` keeps it out of the package. `dist/` is not tracked in git. -- Exit code 0. Blocking failures: 0. -- package_contents, package_secrets, package_binaries, manifest_metadata, readme_metadata, package_dependencies, python_compile, and sensitive_capabilities passed. python_safety and prohibited_financial_activity passed. -- access_domains: the static scan found only `api.diffbot.com`, `kg.diffbot.com`, `llm.diffbot.com`, and `nl.diffbot.com`, which are the `network.domains` entries. -- dependency_vulnerabilities: pinned `dify_plugin==0.11.0` and `requests==2.34.2` had no known vulnerabilities. -- `check-pr-body.py` and `check-pr-review-notes.py` passed on the PR body. -- `test-plugin-install.py` (`INSTALL_METHOD=serverless`) started the plugin and got a response on port 8080. +Marketplace toolkit validator: **TODO(sdk-release): rerun marketplace validator.** The pre-SDK v0.1 numbers that used to be here (31 tests, checksum `6abed6eb…`, `requests==2.34.2` with no known vulnerabilities, a four-host access_domains scan of the plugin's own code, a python_safety pass) were measured on a different dependency set and code path. Do not reuse them. Record dependency_vulnerabilities, access_domains, python_safety, the PR-body checks, and the serverless start-up test from the new run. -Live smoke test, 10 November 2026, each tool class called directly with a real token (not inside Dify): - -```text -[PASS] credential check (2.8s): ok -[PASS] search_web (1.9s): text=Diffbot web search returned 3 result(s). 1. Knowledge Graph | Diffbot (score 0.992) https://www.diffbot.com/products/knowledge-graph/ # Knowledge Graph [](https://www.diffbot.com/products/knowl | json keys=['query', 'timeMs', 'results'] n=3 | first={"title": "Knowledge Graph | Diffbot", "date": null, "pageUrl": "https://www.diffbot.com/products/knowledge-graph/", "score": 0.992} -[PASS] extract (1.9s): text=Diffbot extract for https://www.diffbot.com/company/ About Diffbot | Diffbot (other) https://www.diffbot.com/company/about | json keys=['objects', 'humanLanguage', 'url', 'page_type'] n=1 | first={"title": "About Diffbot | Diffbot", "pageUrl": "https://www.diffbot.com/company/about", "type": "other"} -[PASS] search_organizations (1.5s): text=Organization search returned 2 record(s) of 4. 1. Diffbot — Menlo Park, United States diffbot.com American machine learning and knowledge management company 2. ROS Mobile Robots — Feldkirch, Aus | json keys=['dql', 'hits', 'returned', 'results'] n=2 | first={"name": "Diffbot", "diffbotUri": "http://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ"} -[PASS] search_people (2.4s): employer "Diffbot" + title "CEO" -> 1 public profile (Diffbot's founder and CEO); grouped employments.{...} query -[PASS] search_news (2.1s): text=News search returned 3 article(s) of 3552, newest publication date first. 1. Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein (WinFuture.de — d2026-11-10T19:07) https://winfut | json keys=['dql', 'hits', 'returned', 'results'] n=3 | first={"title": "Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein", "date": "d2026-11-10T19:07", "pageUrl": "https://winfuture.de/news,162580.html"} -[PASS] resolve_entities (1.8s): text=Resolved 4 entities from the supplied text. - Mike Tung [person] id=ESGMaGV9uP0SuTmfPTtNEoA - Menlo Park [location, administrative area, city] id=EQiAUr370NpW1gvx-V4fTrQ - Diffbot [organization] id=EY | json keys=['sentiment', 'entities', 'dql_filter'] n=4 -``` - -A live DQL check decided news ordering. `type:Article tags.label:"Microsoft" date>="2026-01-01"` with `sortBy:date` returned articles from 10 November 2026 first. With `revSortBy:date` it returned 1 January 2026 first. The plugin now uses `sortBy:date`. - -People search with employer `Diffbot` and title `Chief Executive Officer` returned an executive assistant at another company, because separate `employments.*` clauses could match different jobs. The plugin now groups them as `employments.{employer.name:"..." title:"..."}`. Titles still match profile text as written: `CEO` finds Diffbot's founder and `Chief Executive Officer` does not. +Live smoke test: **TODO(sdk-release): rerun.** The 10 November 2026 run covered the pre-SDK package (credential check, web search, extract, organization, people, and news search, and entity resolution). It did not cover crawl. Two behaviors it settled are still in the code. News uses `sortBy:date` (newest first). People search groups employer and title as `employments.{employer.name:"..." title:"..."}` so both clauses must match the same job. Recheck both in the new run. ## Open questions and risks Blockers before a real PR: +- `diffbot` 3.4.0 must be on PyPI and pinned, and the validator rerun on the resulting package. Until then the dependency, access-domain, and safety results are unknown. +- PRIVACY.md does not describe crawl yet. - The package has not been installed in Dify Community Edition or Dify Cloud. Do that, or write the limitation in the PR. - Whoever opens the PR must be allowed to publish under the `diffbot` author name. Open it from a fork owned by, or on behalf of, the `diffbot` organization. +Crawl scope and risk: + +- Crawling user-supplied URLs is the main reason this plugin is high risk. An agent can start a crawl of any public site the user or the model names. The crawl follows links up to the configured limits and runs on Diffbot's infrastructure, against the user's own Diffbot account and quota. The plugin's URL guard stops literal private and metadata targets. It does not resolve hostnames, and it cannot judge whether the user is allowed to crawl a site. That depends on Diffbot's terms and the user's account. +- The `obey_robots` form field defaults to off. The SDK sends `obeyRobots=1` only when it is on. When it is off, nothing is sent and the Diffbot crawler's own robots.txt default applies, whatever that default is. So the toggle can say "off" while the crawler is still obeying robots.txt, or the reverse. Confirm the API default before release and make the label match it. A reviewer may also want robots.txt respected by default in a Dify-facing tool. That is a product call, and the change is a one-line YAML edit. +- Crawl jobs outlive the tool call. A user who starts a large crawl from an agent keeps using quota after the conversation ends, and the plugin has no cleanup. The tool text says the job runs in the background, and `crawl_jobs` can stop it with `delete`. +- `delete` is irreversible and has no confirmation step in the plugin. All that keeps a model from deleting is the LLM description ("only use delete when the user explicitly asks"). It also reaches every crawl job on the token's account, including jobs created outside Dify. If reviewers object, the options are to split delete into its own tool that users can leave disabled, or to limit delete to job names the plugin generated. +- The custom header parameter is sent to every crawled site. The parameter text warns against putting secrets there, but the plugin does not enforce that. + +Credential form: + +- `block_private_hosts` has no observable effect today. No credential or parameter can change the SDK base URLs, so the transport only ever talks to the four public `*.diffbot.com` hosts, and user-supplied URLs are always checked by `validate_public_http_url` whatever the toggle says. The help text ("Leave on unless you run a self-hosted gateway") promises a feature that does not exist. Before submission, remove the field (recommended for the first release) or add a real `api_base` credential that the toggle applies to. The second option also needs a `network.domains` review. + Product and API risks: -- Free tokens may not include Web Search, Knowledge Graph, and Natural Language together. The Account API can succeed while a later call returns 401 or 403. The 10 November live test used one Diffbot token with all of them enabled; a free-plan token has not been tried. -- Organization industry is an exact category string. The MCP server resolves free text through the ontology. v1 will return empty results for "software" when the category is "Software Companies". Either document that sharply or add ontology resolution in a later version. -- Job-title abbreviations are not expanded (`CEO` versus `Chief Executive Officer`). The MCP server does expand them. The tool description tells the model to try both forms, but people search can still under-count until expansion exists. -- Web Search might also accept `token` as a query parameter. This plugin uses only the documented bearer header. If a token type rejects the header, the tool fails closed. That is a live test, not a guess to code around. +- Free tokens may not include Web Search, Knowledge Graph, Natural Language, and Crawl together. The Account API check can pass while a later call returns 401 or 403. No free-plan token has been tested. +- Organization industry is an exact category string. The MCP server resolves free text through the ontology. v1 returns nothing for "software" when the category is "Software Companies". Either document that clearly or add ontology resolution in a later version. +- Job-title abbreviations are not expanded (`CEO` versus `Chief Executive Officer`). The MCP server does expand them. The tool description tells the model to try both forms, but people search can still miss matches until expansion exists. Review and policy risks: -- High risk is the honest box. A reviewer can still ask for a narrower tool set. Dropping Extract would remove user-supplied URL fetch, but people search would still be personal data. Dropping people and entity resolution would look more like Tavily and might be argued as medium, because the remaining URL fetch is still "user-provided URLs" in the medium definition and "proxy or crawl arbitrary URLs" in the high definition. The written rule is to choose the higher label. Do not mark this low because Exa did. Exa's PR text said it does not fetch arbitrary URLs. This plugin does ask Diffbot to fetch them. -- Diffbot's privacy policy says the service is not directed at anyone 18 or under, and gives parents a contact address for privacy requests. The Knowledge Graph can still return a public profile of a younger person, and this plugin cannot filter by age. Call that out in review rather than claiming the plugin is age-safe. -- Dify Cloud customers will send workspace prompts to Diffbot. Whether that needs a DPA or a customer-facing subprocessors note is a Diffbot legal question. The plugin policy links Diffbot's policy and says the plugin itself stores nothing. It does not answer the Dify customer's contract question. +- High risk is the honest box. Extract and crawl both have Diffbot fetch user-supplied URLs, and crawl is the plain "crawl arbitrary URLs" case. A reviewer can still ask for a narrower tool set. Dropping crawl would leave single-page Extract ("user-provided URLs") and people search (personal data), which is still at least medium. The written rule is to pick the higher label. Do not mark this low because Exa did: Exa's PR said it does not fetch arbitrary URLs, and this plugin does. +- Diffbot's privacy policy says the service is not directed at anyone 18 or under, and gives parents a contact address for privacy requests. The Knowledge Graph can still return a public profile of a younger person, and this plugin cannot filter by age. Raise that in review rather than claiming the plugin is age-safe. +- Dify Cloud customers will send workspace prompts and crawl targets to Diffbot. Whether that needs a DPA or a customer-facing subprocessors note is a question for Diffbot legal. - A merged PR might not produce a listing. Watch the Marketplace after merge, as Exa had to. - `meta.minimum_dify_version` is 1.14.0, taken from the SDK compatibility table rather than an install on 1.14.0. If the Community Edition test runs on a newer version and a reviewer asks, either test 1.14.0 or raise the floor to the tested version. From 0690e251e10d24206a2de2273eb3a60e2104da54 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 15:54:37 -1000 Subject: [PATCH 11/15] Remove block_private_hosts credential; keep private-host guard always on Drops the field, _as_bool and the toggle plumbing in GuardedTransport/make_client, and the self-hosted gateway wording in docs. Co-Authored-By: Claude Sonnet 5.5 --- SUBMISSION.md | 10 ++-------- diffbot/README.md | 2 +- diffbot/provider/diffbot.yaml | 8 -------- diffbot/tools/sdk.py | 17 +++-------------- tests/test_sdk_wiring.py | 14 ++++++++------ 5 files changed, 14 insertions(+), 37 deletions(-) diff --git a/SUBMISSION.md b/SUBMISSION.md index 4b5feb9..97914b7 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -35,7 +35,6 @@ Still to do: 3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. 4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. 5. **Update `diffbot/PRIVACY.md` for crawl.** Its "What is sent to Diffbot" table has no rows for seed URLs, crawl patterns, custom headers, or crawl results, and it does not say that crawl jobs and their data stay in the user's Diffbot account until someone deletes them. The privacy text has to match the package before submission. -6. Decide what to do with the `block_private_hosts` credential (see "Open questions and risks"). 7. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. 8. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, crawl seeds, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. @@ -72,7 +71,7 @@ The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches t - `dify_plugin==0.11.0`. - `diffbot==3.4.0`, the official Diffbot Python SDK. **Not pinned yet**: it goes into `requirements.txt` and `pyproject.toml` once 3.4.0 is on PyPI. Until then the tests run against a local editable checkout. - `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. -- Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and, while `block_private_hosts` is on, loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. +- Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. - Calls the SDK does not have yet go through the SDK's public `client.request()`, never its private attributes: token verification (`GET /v4/account?days=1`), crawl data download, and extract options. Each one feature-detects the native SDK method and uses it when present. - The SDK sends the token as a `token` query parameter to api, kg, and nl.diffbot.com, and as `Authorization: Bearer` to llm.diffbot.com (Web Search). Errors are mapped to plugin-safe text and the token is redacted from them. @@ -84,7 +83,6 @@ From `provider/diffbot.yaml` `credentials_for_provider`: | --- | --- | --- | --- | --- | | `diffbot_token` | secret-input | yes | none | Diffbot API token. Format-checked (16 to 256 characters of `A-Za-z0-9._~+-`) and verified with the Account API when the credential is saved. | | `request_timeout` | text-input | no | `30` | Seconds per Diffbot call, 5 to 300. The 300 ceiling is dify_plugin's request limit. | -| `block_private_hosts` | boolean | no | `true` | Toggles the transport host guard described above. See the open question about this field below. | ### Network domains @@ -182,7 +180,7 @@ New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot= - Start a Diffbot crawl job, and list, check, fetch results of, or delete crawl jobs (delete is destructive) - Resolve entities in pasted text to Knowledge Graph ids -Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). The plugin calls the Account API to check the token when the credential is saved. Two optional credential fields set the request timeout (5 to 300 seconds) and a private-host guard. The plugin has no raw DQL tool and takes no email, phone, or IP inputs. +Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). The plugin calls the Account API to check the token when the credential is saved. One optional credential field sets the request timeout (5 to 300 seconds). The private-host guard is always on. The plugin has no raw DQL tool and takes no email, phone, or IP inputs. ## Risk level @@ -332,10 +330,6 @@ Crawl scope and risk: - `delete` is irreversible and has no confirmation step in the plugin. All that keeps a model from deleting is the LLM description ("only use delete when the user explicitly asks"). It also reaches every crawl job on the token's account, including jobs created outside Dify. If reviewers object, the options are to split delete into its own tool that users can leave disabled, or to limit delete to job names the plugin generated. - The custom header parameter is sent to every crawled site. The parameter text warns against putting secrets there, but the plugin does not enforce that. -Credential form: - -- `block_private_hosts` has no observable effect today. No credential or parameter can change the SDK base URLs, so the transport only ever talks to the four public `*.diffbot.com` hosts, and user-supplied URLs are always checked by `validate_public_http_url` whatever the toggle says. The help text ("Leave on unless you run a self-hosted gateway") promises a feature that does not exist. Before submission, remove the field (recommended for the first release) or add a real `api_base` credential that the toggle applies to. The second option also needs a `network.domains` review. - Product and API risks: - Free tokens may not include Web Search, Knowledge Graph, Natural Language, and Crawl together. The Account API check can pass while a later call returns 401 or 403. No free-plan token has been tested. diff --git a/diffbot/README.md b/diffbot/README.md index 549607c..61b7dd2 100644 --- a/diffbot/README.md +++ b/diffbot/README.md @@ -26,7 +26,7 @@ Raw DQL, ontology lookup, Ask, places, and deals are not in this version. Indust 1. Create a token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). 2. In Dify, open Plugins, install this package, and choose Diffbot. 3. Paste the token into **Diffbot token**. -4. Optional: set **Request timeout** (5 to 300 seconds, default 30). **Block private hosts** stays on unless you run a self-hosted gateway. +4. Optional: set **Request timeout** (5 to 300 seconds, default 30). Dify stores the value as a secret. The plugin sends it only to the Diffbot hosts listed in `manifest.yaml`. Web Search uses `Authorization: Bearer`. The other calls use the `token` query parameter, which is how those APIs authenticate. Saving the credential calls `GET https://api.diffbot.com/v4/account` with `days=1` and does not show the account payload. diff --git a/diffbot/provider/diffbot.yaml b/diffbot/provider/diffbot.yaml index 8bac71f..297d650 100644 --- a/diffbot/provider/diffbot.yaml +++ b/diffbot/provider/diffbot.yaml @@ -31,14 +31,6 @@ credentials_for_provider: en_US: "30" help: en_US: Seconds to wait for each Diffbot call, from 5 to 300. Leave blank for 30. - block_private_hosts: - type: boolean - required: false - default: true - label: - en_US: Block private hosts - help: - en_US: Refuse requests to loopback, private, and cloud-metadata addresses. Leave on unless you run a self-hosted gateway. tools: - tools/search_web.yaml - tools/extract.yaml diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py index 2f9b48c..18f569e 100644 --- a/diffbot/tools/sdk.py +++ b/diffbot/tools/sdk.py @@ -61,8 +61,7 @@ class GuardedTransport(httpx.BaseTransport): once an explicit transport is passed, which would break proxied Dify deployments. """ - def __init__(self, inner: httpx.BaseTransport | None = None, *, block_private_hosts: bool = True) -> None: - self._block = block_private_hosts + def __init__(self, inner: httpx.BaseTransport | None = None) -> None: self._proxied: dict[str, httpx.BaseTransport] = {} if inner is None: proxies = urllib.request.getproxies() @@ -81,8 +80,7 @@ def _transport_for(self, url: httpx.URL) -> httpx.BaseTransport: def handle_request(self, request: httpx.Request) -> httpx.Response: if request.url.scheme not in ("http", "https"): raise DiffbotError("Request scheme is not allowed.") - if self._block: - check_public_host(request.url.host or "") + check_public_host(request.url.host or "") return self._transport_for(request.url).handle_request(request) def close(self) -> None: @@ -91,14 +89,6 @@ def close(self) -> None: transport.close() -def _as_bool(value: Any, default: bool) -> bool: - if value is None or value == "": - return default - if isinstance(value, str): - return value.strip().lower() not in {"false", "0", "no", "off"} - return bool(value) - - def parse_timeout(value: Any) -> float: if value is None or (isinstance(value, str) and not value.strip()): return DEFAULT_TIMEOUT @@ -117,8 +107,7 @@ def make_client(credentials: Mapping[str, Any] | None, *, transport: httpx.BaseT raw = credentials.get("diffbot_token", "") token = normalize_token(raw if isinstance(raw, str) else "") timeout = parse_timeout(credentials.get("request_timeout")) - block = _as_bool(credentials.get("block_private_hosts"), True) - return Diffbot(token, timeout=timeout, transport=GuardedTransport(transport, block_private_hosts=block)) + return Diffbot(token, timeout=timeout, transport=GuardedTransport(transport)) def api_error_message(exc: BaseException, token: str) -> str: diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index a7f8609..58abd7b 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -138,12 +138,14 @@ def test_explicit_inner_transport_ignores_proxy_environment(monkeypatch): assert sdk.GuardedTransport(inner)._transport_for(httpx.URL("https://api.diffbot.com/")) is inner -def test_guard_can_be_disabled_for_self_hosted_gateways(): - inner = httpx.MockTransport(lambda r: httpx.Response(200, json={"objects": []})) - client = Diffbot( - TOKEN, analyze_url="http://10.0.0.5/v3", transport=sdk.GuardedTransport(inner, block_private_hosts=False) - ) - assert client.extract("https://example.org") == {"objects": []} +def test_guard_cannot_be_disabled(): + calls = [] + inner = httpx.MockTransport(lambda r: calls.append(r) or httpx.Response(200, json={"objects": []})) + client = Diffbot(TOKEN, analyze_url="http://10.0.0.5/v3", transport=sdk.GuardedTransport(inner)) + with pytest.raises(Exception): + client.extract("https://example.org") + assert calls == [] + assert not hasattr(sdk, "_as_bool") def test_rate_limit_maps_to_a_safe_message(serve): From fd85e348486feb745c9ffba697f1740876858cb4 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 17:59:07 -1000 Subject: [PATCH 12/15] Pin diffbot==3.4.0 and run the SDK tests in CI. The plugin version stays 0.1.0. The editable workspace path is gone. Marketplace submission stays a separate pull request. --- .github/workflows/test.yml | 4 ---- CHANGELOG.md | 13 +++++++++++++ README.md | 6 +++--- SUBMISSION.md | 6 +++--- diffbot/pyproject.toml | 2 +- diffbot/requirements.txt | 6 +----- requirements-dev.txt | 10 +++------- tests/conftest.py | 15 --------------- 8 files changed, 24 insertions(+), 38 deletions(-) create mode 100644 CHANGELOG.md delete mode 100644 tests/conftest.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 8eaec56..ce2791a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -20,8 +20,4 @@ jobs: cache: pip cache-dependency-path: diffbot/requirements.txt - run: pip install -r diffbot/requirements.txt pytest - # TODO(release): the plugin needs the `diffbot` SDK (3.2.0, then 3.3.0), which is not - # published yet and cannot be installed from a private path or unpushed ref here. - # Until then tests/conftest.py skips the SDK-dependent modules. After release, pin - # `diffbot` in diffbot/requirements.txt (installed by the line above) and delete conftest.py. - run: pytest -q diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..8c81047 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,13 @@ +# Changelog + +## 0.1.0 - 2026-11-17 + +Plugin version stays 0.1.0. It is not on the Dify Marketplace. + +### Changed + +- Pin `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. +- Remove the editable `/workspace/diffbot-python` path from `requirements-dev.txt`. +- CI installs the pin from PyPI and runs the SDK-dependent tests (the skip in `tests/conftest.py` is gone). + +This repo does not publish to npm or PyPI. Shipping is a Dify Marketplace pull request, which is separate from merging this branch. Feature-detect fallbacks in `diffbot/tools/sdk.py` are still in the tree for that submission. diff --git a/README.md b/README.md index 1783365..4242260 100644 --- a/README.md +++ b/README.md @@ -33,15 +33,15 @@ Python 3.12+: ```bash python -m venv .venv && . .venv/bin/activate -pip install -r requirements-dev.txt # dify_plugin, pytest, and the local SDK checkout +pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0 pytest -q ``` -The plugin imports the `diffbot` SDK, which is not released at the needed version yet (3.2.0, then 3.3.0). `diffbot/requirements.txt` therefore pins only `dify_plugin==0.11.0` and carries a TODO for the `diffbot` pin. `requirements-dev.txt` installs the local, unpushed SDK checkout editable (`-e /workspace/diffbot-python`); edit that path if your checkout lives elsewhere. +`diffbot/requirements.txt` pins `diffbot==3.4.0`. Until that release is on PyPI, install a local checkout of it editable and do not commit the path. `requirements-dev.txt` does not contain a machine-local path. The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). -GitHub CI (`.github/workflows/test.yml`) cannot see the local path. Until `diffbot` is released it does not install the SDK, and `tests/conftest.py` skips the SDK-dependent modules (the DQL and URL-policy tests still run). The TODOs in both files say what to delete after release. +GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0`, and runs the full suite. ## Package diff --git a/SUBMISSION.md b/SUBMISSION.md index 97914b7..3c39389 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,6 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0` at release) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 is not on PyPI, so the package cannot be built with its real dependency pin. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 is not on PyPI yet, so a marketplace package still cannot be installed from the registry until that SDK release is up. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. Checked on 9 November 2026 against: @@ -26,11 +26,11 @@ Done: - Contact: `support@diffbot.com` in `manifest.yaml`, the plugin README, PRIVACY.md, and the PR body. - Icon: `_assets/icon.png` is the official 2026 Diffbot logomark, the green rounded-square tile (512×512 PNG). - `meta.minimum_dify_version`: 1.14.0. The dify_plugin 0.11.0 README lists Dify 1.14.0 for SDK 0.8.1 and later. The plugin uses no later gated feature (OAuth, dynamic-select, multiple selects, reset_on_change). -- Unit tests: `pytest` reports 109 passed on 17 November 2026, with mocked HTTP, run against a local editable checkout of the unreleased SDK (`requirements-dev.txt`). +- Unit tests: `pytest` reports 109 passed on 17 November 2026, with mocked HTTP, against `diffbot` 3.4.0. GitHub CI installs that pin from PyPI. Still to do: -1. **Release `diffbot` 3.4.0 to PyPI and pin it.** `diffbot/requirements.txt` and `diffbot/pyproject.toml` still carry a `TODO(release)` comment in place of the pin, and that comment names 3.2.0/3.3.0. Replace it with `diffbot==3.4.0` in both files. Delete the `-e /workspace/diffbot-python` line from `requirements-dev.txt`. Then delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options). +1. **Release `diffbot` 3.4.0 to PyPI.** The repo already pins `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`, and `requirements-dev.txt` no longer has an editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. 2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. 3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. 4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. diff --git a/diffbot/pyproject.toml b/diffbot/pyproject.toml index 3c3ee20..991a3d9 100644 --- a/diffbot/pyproject.toml +++ b/diffbot/pyproject.toml @@ -6,5 +6,5 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "dify_plugin==0.11.0", - # TODO(release): add "diffbot==3.2.0", later "diffbot==3.3.0" (not published yet). + "diffbot==3.4.0", ] diff --git a/diffbot/requirements.txt b/diffbot/requirements.txt index f0a99a8..413e758 100644 --- a/diffbot/requirements.txt +++ b/diffbot/requirements.txt @@ -1,6 +1,2 @@ dify_plugin==0.11.0 -# TODO(release): the plugin imports the `diffbot` SDK. Pin it here and in -# pyproject.toml once released: diffbot==3.2.0 (transport, retries, DQL POST), then -# diffbot==3.3.0 (verify_token, crawl_download, extract options, response types). -# Never list both `diffbot` and `diffbot-python`. -# Until then, local/dev runs use requirements-dev.txt (editable local SDK checkout). +diffbot==3.4.0 diff --git a/requirements-dev.txt b/requirements-dev.txt index 67674f8..895cf1a 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,9 +1,5 @@ -# Dev/test-only requirements. Not used for the production package. -# -# Installs the local, unreleased Diffbot Python SDK checkout (editable) so the -# plugin can be exercised against it on this machine. GitHub CI cannot see this -# path and does not use this file; CI stays on mocks until `diffbot` is released. -# TODO: delete the -e line once diffbot>=3.3.0 is on PyPI and pinned in diffbot/requirements.txt. +# Dev/test requirements. Not a second pin: the SDK comes from diffbot/requirements.txt. +# Until diffbot==3.4.0 is on PyPI, install a local 3.4.0 checkout yourself +# (`pip install -e /path/to/diffbot-python`) and do not commit that path. -r diffbot/requirements.txt pytest --e /workspace/diffbot-python diff --git a/tests/conftest.py b/tests/conftest.py deleted file mode 100644 index dbec5fd..0000000 --- a/tests/conftest.py +++ /dev/null @@ -1,15 +0,0 @@ -"""Skip SDK-dependent test modules when the ``diffbot`` SDK is not installed. - -GitHub CI cannot reach the unreleased local SDK (see requirements-dev.txt), so there -only the SDK-independent policy tests run. TODO: once diffbot 3.2.0/3.3.0 is on PyPI, -pin it in diffbot/requirements.txt and delete this file so every test runs in CI. -""" - -try: - # Import a real SDK name: with the repo root on sys.path (``python -m pytest``), - # a bare ``import diffbot`` succeeds on the plugin's own ``diffbot/`` directory - # as a namespace package even when the SDK is missing. - from diffbot import Diffbot # noqa: F401 -except ImportError: - collect_ignore = ["test_tools_http.py", "test_sdk_wiring.py"] - print("diffbot SDK not installed: skipping SDK-dependent tests (see tests/conftest.py)") From 9750da9028e3f0c06e7d89b15caba14a8536a934 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 18:29:13 -1000 Subject: [PATCH 13/15] Correct the submission note now that diffbot==3.4.0 is pinned. The dependencies section still said the pin was not applied. Co-authored-by: Cursor --- SUBMISSION.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SUBMISSION.md b/SUBMISSION.md index 3c39389..cdc0561 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -69,10 +69,10 @@ The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches t ### Dependencies and transport - `dify_plugin==0.11.0`. -- `diffbot==3.4.0`, the official Diffbot Python SDK. **Not pinned yet**: it goes into `requirements.txt` and `pyproject.toml` once 3.4.0 is on PyPI. Until then the tests run against a local editable checkout. +- `diffbot==3.4.0`, the official Diffbot Python SDK, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. It is not on PyPI yet, so tests install a local checkout and that path is not committed. `requirements-dev.txt` has no machine-local path. - `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. - Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. -- Calls the SDK does not have yet go through the SDK's public `client.request()`, never its private attributes: token verification (`GET /v4/account?days=1`), crawl data download, and extract options. Each one feature-detects the native SDK method and uses it when present. +- Token verification, crawl data download, and extract options call the native SDK methods when they exist (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0) and otherwise use public `client.request()`. They never touch private attributes. Those fallbacks stay until the marketplace pull request removes them. - The SDK sends the token as a `token` query parameter to api, kg, and nl.diffbot.com, and as `Authorization: Bearer` to llm.diffbot.com (Web Search). Errors are mapped to plugin-safe text and the token is redacted from them. ### Credential fields From eb059b6c2239f2d103e73379262dc171479a135a Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Tue, 17 Nov 2026 18:38:26 -1000 Subject: [PATCH 14/15] Point diffbot at 3.4.0rc0 for CI verification. --- CHANGELOG.md | 2 +- README.md | 6 +++--- SUBMISSION.md | 12 ++++++------ diffbot/pyproject.toml | 2 +- diffbot/requirements.txt | 2 +- requirements-dev.txt | 3 +-- 6 files changed, 13 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c81047..75fb583 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ Plugin version stays 0.1.0. It is not on the Dify Marketplace. ### Changed -- Pin `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. +- Pin `diffbot==3.4.0rc0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml` so CI can verify the SDK release candidate. Switch to `==3.4.0` for the unison publish. - Remove the editable `/workspace/diffbot-python` path from `requirements-dev.txt`. - CI installs the pin from PyPI and runs the SDK-dependent tests (the skip in `tests/conftest.py` is gone). diff --git a/README.md b/README.md index 4242260..f110d27 100644 --- a/README.md +++ b/README.md @@ -33,15 +33,15 @@ Python 3.12+: ```bash python -m venv .venv && . .venv/bin/activate -pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0 +pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0rc0 pytest -q ``` -`diffbot/requirements.txt` pins `diffbot==3.4.0`. Until that release is on PyPI, install a local checkout of it editable and do not commit the path. `requirements-dev.txt` does not contain a machine-local path. +`diffbot/requirements.txt` currently pins `diffbot==3.4.0rc0` so GitHub CI can install the SDK release candidate from PyPI. Switch this to `==3.4.0` for the unison publish. `requirements-dev.txt` does not contain a machine-local path. The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). -GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0`, and runs the full suite. +GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0rc0`, and runs the full suite. ## Package diff --git a/SUBMISSION.md b/SUBMISSION.md index cdc0561..0ebd034 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,6 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 is not on PyPI yet, so a marketplace package still cannot be installed from the registry until that SDK release is up. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0rc0` for CI verification; switch to `==3.4.0` before the marketplace PR) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0rc0 is on PyPI and is what this branch pins for CI. The marketplace package still needs the stable `3.4.0` pin before that PR is opened. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. Checked on 9 November 2026 against: @@ -30,7 +30,7 @@ Done: Still to do: -1. **Release `diffbot` 3.4.0 to PyPI.** The repo already pins `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`, and `requirements-dev.txt` no longer has an editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. +1. **Release `diffbot` 3.4.0 to PyPI.** This branch currently pins `diffbot==3.4.0rc0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml` so GitHub CI can install the release candidate. Switch the pin to `==3.4.0` before the marketplace pull request. `requirements-dev.txt` has no editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. 2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. 3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. 4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. @@ -69,7 +69,7 @@ The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches t ### Dependencies and transport - `dify_plugin==0.11.0`. -- `diffbot==3.4.0`, the official Diffbot Python SDK, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. It is not on PyPI yet, so tests install a local checkout and that path is not committed. `requirements-dev.txt` has no machine-local path. +- `diffbot==3.4.0rc0`, the official Diffbot Python SDK release candidate, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml` for CI. Switch to `==3.4.0` for the marketplace package. `requirements-dev.txt` has no machine-local path. - `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. - Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. - Token verification, crawl data download, and extract options call the native SDK methods when they exist (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0) and otherwise use public `client.request()`. They never touch private attributes. Those fallbacks stay until the marketplace pull request removes them. @@ -97,7 +97,7 @@ The plugin process only talks to these four hosts. Neither credentials nor param ## Package the file they will attach -From a checkout of the plugin, after `diffbot==3.4.0` is pinned, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): +From a checkout of the plugin, after the SDK pin is set, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): ```bash dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg @@ -172,7 +172,7 @@ Fill this in from the live template. Keep every section. Check exactly one risk ## What changed -New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0`). Eight tools: +New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0rc0` on this branch; `==3.4.0` for the marketplace). Eight tools: - Web search on llm.diffbot.com - Extract a page on api.diffbot.com (auto or a specific page type) @@ -227,7 +227,7 @@ python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./ Checksum: TODO(sdk-release): rerun marketplace validator. Validator: TODO(sdk-release): rerun marketplace validator. Report exit code, blocking failures, and each check, especially: -- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0`. Record what OSV reports, including transitive `httpx`. +- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0rc0` (switch to `==3.4.0` for the marketplace). Record what OSV reports, including transitive `httpx`. - access_domains: expected hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The base URLs now sit inside the SDK, so record what the static scan actually finds. - python_safety: record the result for the new SDK/httpx code (`tools/sdk.py`, crawl tools). diff --git a/diffbot/pyproject.toml b/diffbot/pyproject.toml index 991a3d9..0be5dd6 100644 --- a/diffbot/pyproject.toml +++ b/diffbot/pyproject.toml @@ -6,5 +6,5 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "dify_plugin==0.11.0", - "diffbot==3.4.0", + "diffbot==3.4.0rc0", ] diff --git a/diffbot/requirements.txt b/diffbot/requirements.txt index 413e758..0da08f7 100644 --- a/diffbot/requirements.txt +++ b/diffbot/requirements.txt @@ -1,2 +1,2 @@ dify_plugin==0.11.0 -diffbot==3.4.0 +diffbot==3.4.0rc0 diff --git a/requirements-dev.txt b/requirements-dev.txt index 895cf1a..a4352c1 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,5 +1,4 @@ # Dev/test requirements. Not a second pin: the SDK comes from diffbot/requirements.txt. -# Until diffbot==3.4.0 is on PyPI, install a local 3.4.0 checkout yourself -# (`pip install -e /path/to/diffbot-python`) and do not commit that path. +# SDK pin is in diffbot/requirements.txt (currently the 3.4.0rc0 verification pin). -r diffbot/requirements.txt pytest From 712652849a9f0f8ab2d34fd0f9cfa0e3643902b4 Mon Sep 17 00:00:00 2001 From: Andrew Wessels Date: Wed, 18 Nov 2026 11:59:04 -1000 Subject: [PATCH 15/15] Pin diffbot==3.4.0 (stable, now on PyPI). Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- README.md | 6 +++--- SUBMISSION.md | 12 ++++++------ diffbot/pyproject.toml | 2 +- diffbot/requirements.txt | 2 +- requirements-dev.txt | 1 - 6 files changed, 12 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 75fb583..8c81047 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ Plugin version stays 0.1.0. It is not on the Dify Marketplace. ### Changed -- Pin `diffbot==3.4.0rc0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml` so CI can verify the SDK release candidate. Switch to `==3.4.0` for the unison publish. +- Pin `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. - Remove the editable `/workspace/diffbot-python` path from `requirements-dev.txt`. - CI installs the pin from PyPI and runs the SDK-dependent tests (the skip in `tests/conftest.py` is gone). diff --git a/README.md b/README.md index f110d27..409b8d3 100644 --- a/README.md +++ b/README.md @@ -33,15 +33,15 @@ Python 3.12+: ```bash python -m venv .venv && . .venv/bin/activate -pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0rc0 +pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0 pytest -q ``` -`diffbot/requirements.txt` currently pins `diffbot==3.4.0rc0` so GitHub CI can install the SDK release candidate from PyPI. Switch this to `==3.4.0` for the unison publish. `requirements-dev.txt` does not contain a machine-local path. +`diffbot/requirements.txt` pins `diffbot==3.4.0`, which is on PyPI. `requirements-dev.txt` does not contain a machine-local path. The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). -GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0rc0`, and runs the full suite. +GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0`, and runs the full suite. ## Package diff --git a/SUBMISSION.md b/SUBMISSION.md index 0ebd034..57127f1 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,6 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0rc0` for CI verification; switch to `==3.4.0` before the marketplace PR) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0rc0 is on PyPI and is what this branch pins for CI. The marketplace package still needs the stable `3.4.0` pin before that PR is opened. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 was published to PyPI on 18 November 2026. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. Checked on 9 November 2026 against: @@ -30,7 +30,7 @@ Done: Still to do: -1. **Release `diffbot` 3.4.0 to PyPI.** This branch currently pins `diffbot==3.4.0rc0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml` so GitHub CI can install the release candidate. Switch the pin to `==3.4.0` before the marketplace pull request. `requirements-dev.txt` has no editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. +1. **Release `diffbot` 3.4.0 to PyPI.** The repo already pins `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`, and `requirements-dev.txt` no longer has an editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. 2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. 3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. 4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. @@ -69,7 +69,7 @@ The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches t ### Dependencies and transport - `dify_plugin==0.11.0`. -- `diffbot==3.4.0rc0`, the official Diffbot Python SDK release candidate, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml` for CI. Switch to `==3.4.0` for the marketplace package. `requirements-dev.txt` has no machine-local path. +- `diffbot==3.4.0`, the official Diffbot Python SDK, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. It is on PyPI (published 18 November 2026). `requirements-dev.txt` has no machine-local path. - `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. - Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. - Token verification, crawl data download, and extract options call the native SDK methods when they exist (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0) and otherwise use public `client.request()`. They never touch private attributes. Those fallbacks stay until the marketplace pull request removes them. @@ -97,7 +97,7 @@ The plugin process only talks to these four hosts. Neither credentials nor param ## Package the file they will attach -From a checkout of the plugin, after the SDK pin is set, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): +From a checkout of the plugin, after `diffbot==3.4.0` is pinned, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): ```bash dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg @@ -172,7 +172,7 @@ Fill this in from the live template. Keep every section. Check exactly one risk ## What changed -New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0rc0` on this branch; `==3.4.0` for the marketplace). Eight tools: +New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0`). Eight tools: - Web search on llm.diffbot.com - Extract a page on api.diffbot.com (auto or a specific page type) @@ -227,7 +227,7 @@ python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./ Checksum: TODO(sdk-release): rerun marketplace validator. Validator: TODO(sdk-release): rerun marketplace validator. Report exit code, blocking failures, and each check, especially: -- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0rc0` (switch to `==3.4.0` for the marketplace). Record what OSV reports, including transitive `httpx`. +- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0`. Record what OSV reports, including transitive `httpx`. - access_domains: expected hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The base URLs now sit inside the SDK, so record what the static scan actually finds. - python_safety: record the result for the new SDK/httpx code (`tools/sdk.py`, crawl tools). diff --git a/diffbot/pyproject.toml b/diffbot/pyproject.toml index 0be5dd6..991a3d9 100644 --- a/diffbot/pyproject.toml +++ b/diffbot/pyproject.toml @@ -6,5 +6,5 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "dify_plugin==0.11.0", - "diffbot==3.4.0rc0", + "diffbot==3.4.0", ] diff --git a/diffbot/requirements.txt b/diffbot/requirements.txt index 0da08f7..413e758 100644 --- a/diffbot/requirements.txt +++ b/diffbot/requirements.txt @@ -1,2 +1,2 @@ dify_plugin==0.11.0 -diffbot==3.4.0rc0 +diffbot==3.4.0 diff --git a/requirements-dev.txt b/requirements-dev.txt index a4352c1..7c4b8e6 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,4 +1,3 @@ # Dev/test requirements. Not a second pin: the SDK comes from diffbot/requirements.txt. -# SDK pin is in diffbot/requirements.txt (currently the 3.4.0rc0 verification pin). -r diffbot/requirements.txt pytest