diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c81047..f2379af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## Unreleased (still plugin 0.1.0) + +- Removed the `client.request()` fallbacks in `diffbot/tools/sdk.py` for `verify_token`, `crawl_download`, and extract options, and the local response TypedDicts in `tools/sdk_types.py`. `diffbot==3.4.0` has all of them natively. No behavior change. +- Reran the marketplace validator on the rebuilt package; results are in SUBMISSION.md. + ## 0.1.0 - 2026-11-17 Plugin version stays 0.1.0. It is not on the Dify Marketplace. @@ -10,4 +15,4 @@ Plugin version stays 0.1.0. It is not on the Dify Marketplace. - Remove the editable `/workspace/diffbot-python` path from `requirements-dev.txt`. - CI installs the pin from PyPI and runs the SDK-dependent tests (the skip in `tests/conftest.py` is gone). -This repo does not publish to npm or PyPI. Shipping is a Dify Marketplace pull request, which is separate from merging this branch. Feature-detect fallbacks in `diffbot/tools/sdk.py` are still in the tree for that submission. +This repo does not publish to npm or PyPI. Shipping is a Dify Marketplace pull request, which is separate from merging this branch. The feature-detect fallbacks in `diffbot/tools/sdk.py` were removed afterwards (see below). diff --git a/README.md b/README.md index 409b8d3..aed027c 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ pytest -q `diffbot/requirements.txt` pins `diffbot==3.4.0`, which is on PyPI. `requirements-dev.txt` does not contain a machine-local path. -The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). +The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the pinned SDK (`diffbot==3.4.0`) and its native `verify_token`, `crawl_download`, and extract options. GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0`, and runs the full suite. diff --git a/SUBMISSION.md b/SUBMISSION.md index 57127f1..146a799 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,6 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 was published to PyPI on 18 November 2026. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 18 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. `diffbot` 3.4.0 was published to PyPI on 18 November 2026. The same day the SDK feature-detect fallbacks were removed, the package was rebuilt, and the marketplace validator was rerun on it: 0 blocking failures (results below). The live smoke test against the real API has not been rerun yet. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. Checked on 9 November 2026 against: @@ -26,17 +26,17 @@ Done: - Contact: `support@diffbot.com` in `manifest.yaml`, the plugin README, PRIVACY.md, and the PR body. - Icon: `_assets/icon.png` is the official 2026 Diffbot logomark, the green rounded-square tile (512×512 PNG). - `meta.minimum_dify_version`: 1.14.0. The dify_plugin 0.11.0 README lists Dify 1.14.0 for SDK 0.8.1 and later. The plugin uses no later gated feature (OAuth, dynamic-select, multiple selects, reset_on_change). -- Unit tests: `pytest` reports 109 passed on 17 November 2026, with mocked HTTP, against `diffbot` 3.4.0. GitHub CI installs that pin from PyPI. +- Unit tests: `pytest` with mocked HTTP, run by GitHub CI against `diffbot==3.4.0` from PyPI. +- `diffbot` 3.4.0 is on PyPI (18 November 2026) and pinned. The feature-detect fallbacks in `diffbot/tools/sdk.py` (`verify_token`, `crawl_download`, extract options) and the local TypedDicts in `tools/sdk_types.py` are gone; the plugin calls the native 3.4.0 methods. +- Marketplace validator rerun on the rebuilt package on 18 November 2026: exit 0, 0 blocking failures, serverless start-up test passed. See "Results for the current package". Still to do: -1. **Release `diffbot` 3.4.0 to PyPI.** The repo already pins `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`, and `requirements-dev.txt` no longer has an editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. -2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. -3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. -4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. -5. **Update `diffbot/PRIVACY.md` for crawl.** Its "What is sent to Diffbot" table has no rows for seed URLs, crawl patterns, custom headers, or crawl results, and it does not say that crawl jobs and their data stay in the user's Diffbot account until someone deletes them. The privacy text has to match the package before submission. -7. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. -8. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, crawl seeds, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. +1. **Rerun the live smoke test, including crawl.** Needs a real Diffbot token (none was available when the validator was rerun). The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. +2. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. +3. **Update `diffbot/PRIVACY.md` for crawl.** Its "What is sent to Diffbot" table has no rows for seed URLs, crawl patterns, custom headers, or crawl results, and it does not say that crawl jobs and their data stay in the user's Diffbot account until someone deletes them. The privacy text has to match the package before submission. +4. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. +5. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, crawl seeds, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. ## What the plugin is now @@ -72,7 +72,7 @@ The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches t - `diffbot==3.4.0`, the official Diffbot Python SDK, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. It is on PyPI (published 18 November 2026). `requirements-dev.txt` has no machine-local path. - `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. - Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. -- Token verification, crawl data download, and extract options call the native SDK methods when they exist (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0) and otherwise use public `client.request()`. They never touch private attributes. Those fallbacks stay until the marketplace pull request removes them. +- Token verification, crawl data download, and extract options call the native SDK methods (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0). The old `client.request()` fallbacks were removed on 18 November 2026. - The SDK sends the token as a `token` query parameter to api, kg, and nl.diffbot.com, and as `Authorization: Bearer` to llm.diffbot.com (Web Search). Errors are mapped to plugin-safe text and the token is redacted from them. ### Credential fields @@ -97,7 +97,7 @@ The plugin process only talks to these four hosts. Neither credentials nor param ## Package the file they will attach -From a checkout of the plugin, after `diffbot==3.4.0` is pinned, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): +From a checkout of the plugin, with the Dify plugin CLI (the 18 November 2026 run used `dify` v0.6.15, the same as the v0.1 run): ```bash dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg @@ -105,7 +105,7 @@ dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report ``` -The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below has `TODO(sdk-release)` placeholders where the new run's summary goes. +The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit) (it needs `yq` on `PATH`). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below already carries the 18 November 2026 run's results. `dify plugin package` produces an unsigned archive. Dify signs marketplace packages after merge. A third-party signature is only for Community Edition installs outside the marketplace: @@ -152,7 +152,7 @@ Later versions add a new file such as `diffbot/diffbot/diffbot-0.2.0.difypkg` an ## PR body to paste -Fill this in from the live template. Keep every section. Check exactly one risk box: none, or more than one, gets the `risk: missing` label. Do not paste it as-is. Every `TODO(sdk-release)` line has to be replaced with real results first. +Fill this in from the live template. Keep every section. Check exactly one risk box: none, or more than one, gets the `risk: missing` label. Do not paste it as-is. The live smoke test and the Community Edition / Cloud install lines are still `PENDING` and have to be replaced with real results first. If the package is rebuilt, rerun the validator and update the checksum. ````markdown # Plugin Submission @@ -219,21 +219,22 @@ Extract and crawl both send user-supplied http(s) URLs to Diffbot, and Diffbot f ## Local validation ```text -dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI +dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI v0.6.15 dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report --pr-body-file pr-body.md ``` -Checksum: TODO(sdk-release): rerun marketplace validator. +Checksum (`dify plugin checksum`): `7c9012578f4a2a98b02b0693fe484962de3305620dd0fb63b2de165bb85e3e91`. Archive: 48,816 bytes, 32 files. -Validator: TODO(sdk-release): rerun marketplace validator. Report exit code, blocking failures, and each check, especially: -- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0`. Record what OSV reports, including transitive `httpx`. -- access_domains: expected hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The base URLs now sit inside the SDK, so record what the static scan actually finds. -- python_safety: record the result for the new SDK/httpx code (`tools/sdk.py`, crawl tools). +Validator (dify-marketplace-toolkit `83ee707`): exit 0, 0 blocking failures, 0 execution failures. All 12 checks PASS: package_contents, package_secrets, package_binaries, manifest_metadata, readme_metadata, package_dependencies, python_compile, sensitive_capabilities (blocking), and python_safety, prohibited_financial_activity, access_domains, dependency_vulnerabilities (warning). Warnings, explained: +- dependency_vulnerabilities: `requirements.txt` has 2 pinned, 0 unpinned (`dify_plugin==0.11.0`, `diffbot==3.4.0`). OSV reports no known vulnerabilities in the pinned set. +- access_domains: "manifest declares network.domains but the static scan found no literal hostname; this is expected for SDK-based plugins." The Diffbot base URLs live inside the `diffbot` SDK. The plugin cannot change them, and the four declared hosts are the only ones the SDK calls. +- package_secrets: `tools/sdk.py` `INVALID_TOKEN` was flagged for review. It is the error text "Invalid or unauthorized Diffbot token.", not a secret. +- python_safety: `tools/sdk.py` returns or raises values that may contain credentials (the `Diffbot(...)` client built with the user's token, and error messages). Every error string passes through `redact(..., token)` first, and the client object never reaches tool output. -Unit tests: `pytest` 109 passed (mocked HTTP). Plugin start-up (`validator/test-plugin-install.py`, `INSTALL_METHOD=serverless`): TODO(sdk-release): rerun marketplace validator. +Unit tests: `pytest` with mocked HTTP, green in GitHub CI on `diffbot==3.4.0`. Plugin start-up (`validator/test-plugin-install.py`, `INSTALL_METHOD=serverless`): success ("Serving Flask app", request to port 8080 succeeded). -Live smoke test: TODO(sdk-release): rerun every tool, including a crawl start/status/results/delete cycle, against the Diffbot API with a real token. +Live smoke test: PENDING. Rerun every tool, including a crawl start/status/results/delete cycle, against the Diffbot API with a real token. ## Reviewer notes @@ -300,24 +301,19 @@ The SDK handles authentication as the API docs describe. Extract, Account, Crawl ## Results for the current package -Unit tests, no network, 17 November 2026, against a local editable checkout of the unreleased SDK: +Unit tests: no network, mocked HTTP, run by GitHub CI against `diffbot==3.4.0` from PyPI. They now cover only the native SDK methods; the "legacy SDK" parametrization went away with the fallbacks. -```text -pytest -109 passed -``` - -Package, checksum, archive size and file list: **TODO(sdk-release): rerun marketplace validator.** The package should hold the manifest, provider, eight tool YAML/Python pairs, the shared modules (`client.py`, `common.py`, `sdk.py`, `sdk_types.py`, `dql_builder.py`, `shape.py`), README, PRIVACY.md, icon, `main.py`, `requirements.txt`, and `pyproject.toml`. Check the real archive against that list. `.env.example` stays in git for remote debugging and `.difyignore` keeps it out of the package. `dist/` is not tracked in git. +Package, 18 November 2026, `dify` CLI v0.6.15, built from the commit that removed the fallbacks: `dist/diffbot-0.1.0.difypkg`, 48,816 bytes, 32 files, sha256 `018d6e6a7951cb1dd033ff6fcb1f2f36734fc7c543f6fb6fe6faf0d6ba74267e`, `dify plugin checksum` `7c9012578f4a2a98b02b0693fe484962de3305620dd0fb63b2de165bb85e3e91`. Contents: `manifest.yaml`, `main.py`, `requirements.txt`, `pyproject.toml`, `README.md`, `PRIVACY.md`, `_assets/icon.png`, `provider/` (`__init__.py`, `diffbot.py`, `diffbot.yaml`), eight tool YAML/Python pairs, and the shared modules `tools/__init__.py`, `client.py`, `common.py`, `sdk.py`, `dql_builder.py`, `shape.py`. No `.env`, `__pycache__`, tests, or `dist/`. `.env.example` stays in git for remote debugging, and `.difyignore` keeps it out of the package. `dist/` is not tracked in git. -Marketplace toolkit validator: **TODO(sdk-release): rerun marketplace validator.** The pre-SDK v0.1 numbers that used to be here (31 tests, checksum `6abed6eb…`, `requests==2.34.2` with no known vulnerabilities, a four-host access_domains scan of the plugin's own code, a python_safety pass) were measured on a different dependency set and code path. Do not reuse them. Record dependency_vulnerabilities, access_domains, python_safety, the PR-body checks, and the serverless start-up test from the new run. +Marketplace toolkit validator (`langgenius/dify-marketplace-toolkit` at `83ee707`, run with `--pr-body-file` on the PR body above): exit 0, 0 blocking failures, 4 warning categories (each one is explained in the PR body's "Local validation" section), 0 execution failures. sensitive_capabilities passed against the PR body. Serverless start-up test: success. The pre-SDK v0.1 numbers (31 tests, checksum `6abed6eb…`, `requests==2.34.2`) describe a different package. Do not reuse them. -Live smoke test: **TODO(sdk-release): rerun.** The 10 November 2026 run covered the pre-SDK package (credential check, web search, extract, organization, people, and news search, and entity resolution). It did not cover crawl. Two behaviors it settled are still in the code. News uses `sortBy:date` (newest first). People search groups employer and title as `employments.{employer.name:"..." title:"..."}` so both clauses must match the same job. Recheck both in the new run. +Live smoke test: **PENDING.** It needs a real Diffbot token, and none was available when the validator was rerun. The 10 November 2026 run covered the pre-SDK package (credential check, web search, extract, organization, people, and news search, and entity resolution). It did not cover crawl. Two behaviors it settled are still in the code. News uses `sortBy:date` (newest first). People search groups employer and title as `employments.{employer.name:"..." title:"..."}` so both clauses must match the same job. Recheck both in the new run. ## Open questions and risks Blockers before a real PR: -- `diffbot` 3.4.0 must be on PyPI and pinned, and the validator rerun on the resulting package. Until then the dependency, access-domain, and safety results are unknown. +- The live smoke test (every tool, including a crawl cycle) has not been rerun on the 3.4.0 package. - PRIVACY.md does not describe crawl yet. - The package has not been installed in Dify Community Edition or Dify Cloud. Do that, or write the limitation in the PR. - Whoever opens the PR must be allowed to publish under the `diffbot` author name. Open it from a fork owned by, or on behalf of, the `diffbot` organization. diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py index 18f569e..3875183 100644 --- a/diffbot/tools/sdk.py +++ b/diffbot/tools/sdk.py @@ -1,22 +1,13 @@ """Builds the Diffbot SDK client and adapts it to this plugin's policy. -The plugin runs on the ``diffbot`` Python SDK (sync ``Diffbot`` client). Everything -the SDK does not expose yet is reached through the public ``client.request()`` -transport, never through private attributes, and each such helper feature-detects -the real method so it drops out when the SDK ships it: - -* ``verify_token`` -> diffbot 3.3.0 (fallback: ``GET /v4/account?days=1``) -* ``crawl_download`` -> diffbot 3.3.0 (fallback: ``GET /v3/crawl/data``) -* extract options -> diffbot 3.3.0 (fallback: ``GET /v3/{api}`` with the same params) - -TODO: when diffbot 3.2.0 / 3.3.0 are released, pin ``diffbot`` in -``requirements.txt`` / ``pyproject.toml`` and delete the fallbacks. +The plugin runs on the ``diffbot`` Python SDK (sync ``Diffbot`` client), pinned to +``diffbot==3.4.0``. Every call goes through the guarded transport built here, and SDK +errors are mapped to token-free plugin messages. """ from __future__ import annotations import csv -import inspect import io import json import urllib.request @@ -36,6 +27,7 @@ RateLimitError, ValidationError, ) +from diffbot.types import CrawlJob, ExtractResponse from tools.client import ( DiffbotError, @@ -43,7 +35,6 @@ normalize_token, redact, ) -from tools.sdk_types import CrawlJob, ExtractResponse DEFAULT_TIMEOUT = 30.0 MIN_TIMEOUT = 5.0 @@ -161,70 +152,15 @@ def verify_token(client: Diffbot) -> bool: Only an auth failure means False; other errors propagate. """ - native = getattr(client, "verify_token", None) - if callable(native): - return bool(native()) - try: - client.request("GET", f"{client.api_base}/v4/account", params={"token": client.token, "days": 1}) - except AuthError: - return False - return True + return client.verify_token() # ---------------------------------------------------------------- extract -EXTRACT_OPTION_WIRE = { - "timeout": "timeout", - "mode": "mode", - "fields": "fields", - "fallback": "fallback", - "discussion": "discussion", - "paging": "paging", - "max_pages": "maxPages", - "natural_language": "naturalLanguage", - "summary_num_sentences": "summaryNumSentences", - "max_tags": "maxTags", - "tag_confidence": "tagConfidence", - "category_confidence": "categoryConfidence", - "use_proxy": "useProxy", -} -_LIST_OPTIONS = {"fields", "natural_language"} - - -def _supports_extract_options(client: Diffbot) -> bool: - try: - params = inspect.signature(client.extract).parameters - except (TypeError, ValueError): - return False - return "mode" in params or any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()) - - -def _wire_value(value: Any) -> Any: - if isinstance(value, bool): - return "true" if value else "false" - if isinstance(value, (list, tuple)): - return ",".join(str(item) for item in value) - return value - - def extract(client: Diffbot, url: str, api: str, fmt: str, options: Mapping[str, Any]) -> ExtractResponse: - """Extract ``url``. ``options`` use the SDK 3.3.0 keyword names (unset keys omitted).""" + """Extract ``url``. ``options`` use the SDK's keyword names (unset keys omitted).""" options = {key: value for key, value in options.items() if value is not None} - if not options or _supports_extract_options(client): - return cast(ExtractResponse, client.extract(url, api, fmt, **options)) - # Fallback for SDKs without extract options: same request the SDK builds. - params: dict[str, Any] = {"token": client.token, "url": url, "timeout": 30000} - if fmt == "markdown": - params["mode"] = "llm" - for key, value in options.items(): - params[EXTRACT_OPTION_WIRE[key]] = _wire_value(value) - timeout = params.get("timeout") - per_call = (float(timeout) / 1000.0 + 10.0) if isinstance(timeout, (int, float)) else None - response = client.request("GET", f"{client.analyze_url}/{api}", params=params, timeout=per_call) - data = response.json() - if isinstance(data, dict) and "errorCode" in data: - raise ExtractionError(data["errorCode"], data.get("error", "")) - return cast(ExtractResponse, data) + return cast(ExtractResponse, client.extract(url, api, fmt, **options)) # ---------------------------------------------------------------- crawl @@ -246,48 +182,23 @@ def crawl_get_job(client: Diffbot, job_name: str) -> CrawlJob: return cast(CrawlJob, job) -def _download_text(result: Any) -> Any: - """Coerce a ``crawl_download`` return value (Response, bytes, str, parsed) to text/objects.""" - if isinstance(result, (list, dict)): - return result - if hasattr(result, "text"): - return result.text - if isinstance(result, (bytes, bytearray)): - return bytes(result).decode("utf-8", errors="replace") - return str(result) - - def crawl_download( client: Diffbot, job_name: str, *, format: str = "json", type: str | None = None, num: int | None = None ) -> Any: - """Fetch crawl output as parsed rows/objects (a list, or a dict for JSON objects).""" - if type == "urls": - format = "csv" # the URL report is CSV; the JSON decoder would reject it - native = getattr(client, "crawl_download", None) - if callable(native): - raw = native(job_name, format=format, type=type, num=num) - else: - params: dict[str, Any] = {"token": client.token, "name": job_name} - if format and format != "json": - params["format"] = format - if type: - params["type"] = type - if num is not None: - params["num"] = num - raw = client.request("GET", f"{client.crawler_url}/data", params=params, follow_redirects=True) - data = _download_text(raw) + """Fetch crawl output as parsed rows/objects. CSV (and the ``urls`` report, which is + always CSV) is parsed to rows; for JSON the SDK raises ``JobError`` on an error body.""" + data = client.crawl_download(job_name, format=format, type=type, num=num) # type: ignore[call-overload] if not isinstance(data, str): return data - if not data.strip(): - return [] - try: - parsed = json.loads(data) - except ValueError: - return list(csv.DictReader(io.StringIO(data))) - if isinstance(parsed, dict) and parsed.get("errorCode"): - # Same as diffbot 3.3.0: e.g. an unknown job is HTTP 200 + errorCode. - raise JobError(job_name, f"Crawl data download failed: {parsed.get('error') or 'unexpected response'}") - return parsed + if data.lstrip().startswith("{"): + # e.g. an unknown job is HTTP 200 + errorCode even when CSV was asked for + try: + parsed = json.loads(data) + except ValueError: + parsed = None + if isinstance(parsed, dict) and parsed.get("errorCode"): + raise JobError(job_name, f"Crawl data download failed: {parsed.get('error') or 'unexpected response'}") + return list(csv.DictReader(io.StringIO(data))) if data.strip() else [] def as_dict(value: Any, what: str) -> dict[str, Any]: diff --git a/diffbot/tools/sdk_types.py b/diffbot/tools/sdk_types.py deleted file mode 100644 index 2287883..0000000 --- a/diffbot/tools/sdk_types.py +++ /dev/null @@ -1,37 +0,0 @@ -"""Response shapes the plugin reads from the Diffbot SDK. - -TODO(diffbot 3.3.0): the SDK ships response TypedDicts in ``diffbot.types``. Until -the pin requires it, these local ``total=False`` equivalents describe only the -fields this plugin touches. Delete the fallback once ``diffbot>=3.3.0`` is pinned. -""" - -from __future__ import annotations - -from typing import Any, TypedDict - -try: # diffbot >= 3.3.0 ships these; prefer them. - from diffbot.types import CrawlJob, ExtractResponse # type: ignore[attr-defined] - - SDK_TYPES = True -except ImportError: # diffbot 3.2.x: local equivalents below - SDK_TYPES = False - - -if not SDK_TYPES: - - class ExtractResponse(TypedDict, total=False): # type: ignore[no-redef] - objects: list[dict[str, Any]] - humanLanguage: str - errorCode: int - error: str - - class CrawlJobStatus(TypedDict, total=False): - status: int - message: str - - class CrawlJob(TypedDict, total=False): # type: ignore[no-redef] - name: str - jobStatus: CrawlJobStatus - pageCrawlSuccesses: int - pageProcessSuccesses: int - objectsFound: int diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py index 58abd7b..ee4a568 100644 --- a/tests/test_sdk_wiring.py +++ b/tests/test_sdk_wiring.py @@ -1,4 +1,4 @@ -"""SDK wiring: credential check, host guard, extract options, crawl tools, feature detection. +"""SDK wiring: credential check, host guard, extract options, crawl tools. All HTTP is served by httpx.MockTransport; no live Diffbot calls. """ @@ -23,22 +23,6 @@ CREDS = {"diffbot_token": TOKEN} -@pytest.fixture(autouse=True, params=["native", "legacy"]) -def sdk_flavor(request, monkeypatch): - """Run every test against the SDK as installed (3.3.0 methods) and against an SDK - without them, which exercises the plugin's client.request() fallbacks.""" - if request.param == "legacy": - monkeypatch.delattr(Diffbot, "verify_token") - monkeypatch.delattr(Diffbot, "crawl_download") - real_extract = Diffbot.extract - - def old_extract(self, url, api="analyze", fmt="markdown"): - return real_extract(self, url, api, fmt) - - monkeypatch.setattr(Diffbot, "extract", old_extract) - return request.param - - @pytest.fixture def serve(monkeypatch): """Install ``handler(request) -> httpx.Response``; returns the list of seen requests.""" @@ -100,14 +84,6 @@ def test_request_timeout_is_applied(): assert client._http.timeout.read == 45.0 # test-only peek at the httpx client -def test_native_verify_token_is_preferred(): - class Fake: - def verify_token(self): - return False - - assert sdk.verify_token(Fake()) is False # type: ignore[arg-type] - - # ------------------------------------------------------------ host guard @@ -300,7 +276,7 @@ def test_crawl_jobs_status_for_missing_job(serve): _jobs({"action": "status", "job_name": "nope"}) -def test_crawl_results_fall_back_to_the_data_endpoint_and_parse_csv(serve): +def test_crawl_url_report_reads_the_data_endpoint_and_parses_csv(serve): csv_text = 'Url,Crawl Status\nhttps://a.example.org,Success\nhttps://b.example.org,Success\n' seen = serve(lambda r: httpx.Response(200, text=csv_text)) payload = _json(_jobs({"action": "results", "job_name": "j1", "result_type": "urls", "limit": 1})) @@ -311,16 +287,6 @@ def test_crawl_results_fall_back_to_the_data_endpoint_and_parse_csv(serve): assert payload["results"][0]["Url"] == "https://a.example.org" -def test_crawl_results_use_native_crawl_download_when_available(): - class Fake: - token = TOKEN - - def crawl_download(self, name, *, format="json", type=None, num=None): - return httpx.Response(200, json=[{"title": "t"}]) - - assert sdk.crawl_download(Fake(), "j", num=5) == [{"title": "t"}] # type: ignore[arg-type] - - def test_crawl_download_job_error_shows_the_reason(serve): serve(lambda r: httpx.Response(200, json={"errorCode": 500, "error": "No collection record"})) with pytest.raises(DiffbotError, match="No collection record"):