diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..8c81047 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,13 @@ +# Changelog + +## 0.1.0 - 2026-11-17 + +Plugin version stays 0.1.0. It is not on the Dify Marketplace. + +### Changed + +- Pin `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. +- Remove the editable `/workspace/diffbot-python` path from `requirements-dev.txt`. +- CI installs the pin from PyPI and runs the SDK-dependent tests (the skip in `tests/conftest.py` is gone). + +This repo does not publish to npm or PyPI. Shipping is a Dify Marketplace pull request, which is separate from merging this branch. Feature-detect fallbacks in `diffbot/tools/sdk.py` are still in the tree for that submission. diff --git a/README.md b/README.md index b4f8a4e..409b8d3 100644 --- a/README.md +++ b/README.md @@ -14,26 +14,34 @@ The installable plugin is in [`diffbot/`](diffbot/). Its [README](diffbot/README | --- | --- | | Web search | `GET https://llm.diffbot.com/api/v1/web_search` (bearer token) | | Extract page | `GET https://api.diffbot.com/v3/{analyze\|article\|product\|discussion\|image\|video\|list\|event}` (`token` query parameter) | -| Search organizations | `GET https://kg.diffbot.com/kg/v3/dql` | +| Search organizations | `POST https://kg.diffbot.com/kg/v3/dql` | | Search people | same DQL endpoint, public professional profiles only | | Search news | same DQL endpoint, newest publication date first via `sortBy:date` | -| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` | +| Start crawl | `GET https://api.diffbot.com/v3/crawl` (create only; returns a job name) | +| Crawl jobs | list / status / results / **delete (destructive)** on `/v3/crawl` and `/v3/crawl/data` | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Saving the credential checks `GET https://api.diffbot.com/v4/account?days=1`. Get a free token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). -Crawl, raw DQL, ontology lookup, places, and deals are not in v0.1. See SUBMISSION.md for why. +Raw DQL, ontology lookup, Ask, places, and deals are not in this version. See SUBMISSION.md for why. + +All HTTP goes through the [Diffbot Python SDK](https://github.com/diffbot/diffbot-python) (`diffbot.Diffbot`), behind a transport that refuses private, loopback, and cloud-metadata hosts. ## Run the tests -Python 3.12: +Python 3.12+: ```bash -pip install -r diffbot/requirements.txt pytest -pytest +python -m venv .venv && . .venv/bin/activate +pip install -r requirements-dev.txt # dify_plugin, pytest, and diffbot==3.4.0 +pytest -q ``` -The tests mock HTTP and make no Diffbot calls. `diffbot/requirements.txt` pins `dify_plugin==0.11.0` and `requests==2.34.2`. +`diffbot/requirements.txt` pins `diffbot==3.4.0`, which is on PyPI. `requirements-dev.txt` does not contain a machine-local path. + +The tests serve every request from `httpx.MockTransport` and make no Diffbot calls. The wiring tests run against the SDK as installed and against a copy with the 3.3.0 methods removed, which covers the plugin's `client.request()` fallbacks (`verify_token`, `crawl_download`, extract options). + +GitHub CI (`.github/workflows/test.yml`) installs `diffbot/requirements.txt`, including `diffbot==3.4.0`, and runs the full suite. ## Package diff --git a/SUBMISSION.md b/SUBMISSION.md index 6f2be4a..57127f1 100644 --- a/SUBMISSION.md +++ b/SUBMISSION.md @@ -1,6 +1,7 @@ # Publishing the Diffbot plugin to the Dify Marketplace -This is the runbook for publishing the plugin from the Diffbot organization. As of 10 November 2026 the source is public at https://github.com/diffbot/dify-diffbot and the package passes the marketplace validator, but no pull request has been opened against `langgenius/dify-plugins` and the package has not been installed in Dify Community Edition or Dify Cloud. +This is the runbook for publishing the plugin from the Diffbot organization. The source is public at https://github.com/diffbot/dify-diffbot. The plugin described here is the `feat/sdk-3.x-wiring` branch: eight tools, including crawl, built on the `diffbot` Python SDK (pinned to `diffbot==3.4.0`) and httpx instead of `requests`. As of 17 November 2026 no pull request has been opened against `langgenius/dify-plugins`, and this version has not been installed in Dify Community Edition or Dify Cloud. It also has not been through the marketplace validator yet. `diffbot` 3.4.0 was published to PyPI on 18 November 2026. Every validator result in this file that depends on that is marked `TODO(sdk-release): rerun marketplace validator`. The pre-SDK v0.1 results that used to be here no longer describe this package and have been removed. + Checked on 9 November 2026 against: @@ -15,6 +16,7 @@ Checked on 9 November 2026 against: - Marketplace search API `POST https://marketplace.dify.ai/api/v1/plugins/search/advanced` and `POST https://marketplace.dify.ai/api/v1/plugins/batch` - Diffbot docs at docs.diffbot.com and the public tool list in [diffbot/diffbot-mcp](https://github.com/diffbot/diffbot-mcp) + ## Status before a PR Done: @@ -24,17 +26,78 @@ Done: - Contact: `support@diffbot.com` in `manifest.yaml`, the plugin README, PRIVACY.md, and the PR body. - Icon: `_assets/icon.png` is the official 2026 Diffbot logomark, the green rounded-square tile (512×512 PNG). - `meta.minimum_dify_version`: 1.14.0. The dify_plugin 0.11.0 README lists Dify 1.14.0 for SDK 0.8.1 and later. The plugin uses no later gated feature (OAuth, dynamic-select, multiple selects, reset_on_change). -- Live API check: every tool was called against the real Diffbot API on 10 November 2026. Results are below. +- Unit tests: `pytest` reports 109 passed on 17 November 2026, with mocked HTTP, against `diffbot` 3.4.0. GitHub CI installs that pin from PyPI. Still to do: -1. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. -2. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. -3. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. +1. **Release `diffbot` 3.4.0 to PyPI.** The repo already pins `diffbot==3.4.0` in `diffbot/requirements.txt` and `diffbot/pyproject.toml`, and `requirements-dev.txt` no longer has an editable checkout path. Still delete the feature-detect fallbacks in `diffbot/tools/sdk.py` that 3.4.0 makes unnecessary (`verify_token`, `crawl_download`, extract options) before the marketplace pull request. Do not open that marketplace PR until the SDK is on PyPI and the validator has been rerun. +2. **TODO(sdk-release): rerun marketplace validator.** Repackage, record the new checksum, and run the toolkit validator with `--pr-body-file` and the OSV lookup. Fill in the "Local validation" and "Results for the current package" sections below with what it reports. +3. **Rerun the live smoke test, including crawl.** The 10 November 2026 live run was against the pre-SDK package and did not cover crawl. Run every tool against the real API, including `crawl_start` → `crawl_jobs` status → results → delete on a small job whose name is 30 characters or fewer. +4. Install the `.difypkg` on current Dify Community Edition and on Dify Cloud and record the version and result in the PR. The checklist asks for that test or a written limitation. Remote debugging (`python -m main` with a debug key from the Dify plugin page) is the quickest path. +5. **Update `diffbot/PRIVACY.md` for crawl.** Its "What is sent to Diffbot" table has no rows for seed URLs, crawl patterns, custom headers, or crawl results, and it does not say that crawl jobs and their data stay in the user's Diffbot account until someone deletes them. The privacy text has to match the package before submission. +7. Confirm that Diffbot is fine with its logo on the Dify Marketplace listing. Reviewers can ask for proof of the right to use a mark. +8. Have someone at Diffbot read `diffbot/PRIVACY.md` against the current Diffbot privacy policy and terms: user queries, page URLs, crawl seeds, and pasted text are sent to Diffbot, and organization, people, and news results can include personal data already on the public web. + +## What the plugin is now + +### Tools + +`provider/diffbot.yaml` registers eight tools, one YAML and one Python file each under `diffbot/tools/`: + +| Tool | Diffbot host | What it does | +| --- | --- | --- | +| `search_web` | llm.diffbot.com | Web search. 1 to 50 results (default 5), optional snippet length and token budget. A query that is only `url:` returns the indexed copy of that page. | +| `extract` | api.diffbot.com | Diffbot fetches one public page and returns structured content (auto/analyze, article, product, discussion, image, video, list, event), as markdown or JSON, with optional Extract API options. | +| `search_organizations` | kg.diffbot.com | Knowledge Graph DQL built from typed fields: name, homepage, industry category, location, headcount and revenue bounds, public-company status. Up to 100 records. | +| `search_people` | kg.diffbot.com | Knowledge Graph search for people with a public professional presence, by name, employer, title, location. Up to 100 records. | +| `search_news` | kg.diffbot.com | Knowledge Graph article search by mentioned entity, phrase, publisher, language, and date, newest first. Up to 100 records. | +| `crawl_start` | api.diffbot.com (`/v3/crawl`) | Starts a background Diffbot crawl job on 1 to 20 public seed URLs and returns only the job name. | +| `crawl_jobs` | api.diffbot.com (`/v3/crawl`, `/v3/crawl/data`) | `list`, `status`, `results` (up to 1,000 rows, flagged when truncated), or `delete`. Delete is destructive. | +| `resolve_entities` | nl.diffbot.com | Finds entities in up to 20,000 characters of pasted text and links them to Knowledge Graph ids, with sentiment. | + +The `enhance_entity` tool was removed (commit 7323b7b) so the tool set matches the Diffbot Python SDK surface. The plugin has no raw DQL tool. + +### Crawl support + +- **Background jobs.** `crawl_start` always calls the SDK with `watch=False` and returns as soon as Diffbot reports the job name. Polling would run past the plugin's 300-second call limit. The job keeps running on Diffbot after the Dify call returns, and it uses the account's crawl quota until it finishes or someone deletes it. The model is told to check `crawl_jobs` `status` and then fetch `results`. +- **Job names, 30 characters max.** Optional `job_name` may use letters, digits, `-`, `_`, and `.`, up to 30 characters. Leave it empty and Diffbot generates `crawl-`. The plugin checks the length itself because the crawler answers a longer name with HTTP 200 plus `errorCode` 505 and creates nothing. +- **Bounds.** Up to 20 seeds. `max_to_crawl` and `max_to_process` are 1 to 10,000 (default 100). `hops` is 0 to 10 (default 2). `restrict_domain` is on by default. `crawl_delay` is −1 to 3,600 seconds (−1 means the Diffbot default). Optional URL crawl and process patterns, `obey_robots` (form default off; see the robots.txt note under risks), `use_proxies` (default off), a custom request header, and an optional `api_url` for per-page extraction. +- **Delete is destructive.** `crawl_jobs` `delete` permanently removes the job and the data it collected. There is no undo. The tool, action label, and LLM description all say so, and the LLM description tells the model to delete only when the user explicitly asks. The plugin itself has no confirmation step, so an agent that ignores that instruction can still delete a job. +- **Account scope.** `list`, `status`, `results`, and `delete` work on every crawl job owned by the token's Diffbot account, including jobs made outside Dify. Nothing limits a Dify workspace to the jobs it started. +- **Results.** Up to 1,000 rows, string cells clipped to 500 characters. The plugin asks for one extra row so it can set `truncated`. `result_type: urls` returns the crawled-URL report (CSV, parsed to rows). + +### Dependencies and transport + +- `dify_plugin==0.11.0`. +- `diffbot==3.4.0`, the official Diffbot Python SDK, pinned in `diffbot/requirements.txt` and `diffbot/pyproject.toml`. It is on PyPI (published 18 November 2026). `requirements-dev.txt` has no machine-local path. +- `httpx` is the HTTP client. It comes in through the SDK (`httpx>=0.27.0`, along with the SDK's `click` and `rich`) and is not pinned directly by the plugin. `requests` is gone: it is no longer imported or listed, and the old `requests==2.34.2` pin was dropped. +- Every SDK call goes through `GuardedTransport` (`tools/sdk.py`). It is an httpx transport that refuses non-http(s) schemes and loopback, private, link-local, reserved, multicast, `localhost`, `*.local`/`*.internal`, and cloud-metadata hosts before a socket opens. The guard is always on; there is no credential to turn it off, and the plugin does not support self-hosted gateways or custom base URLs. It also honors `HTTP(S)_PROXY`/`ALL_PROXY`/`NO_PROXY` like the old `requests` client did, so proxied Dify deployments still work. +- Token verification, crawl data download, and extract options call the native SDK methods when they exist (`verify_token`, `crawl_download`, and `extract` options, all in 3.4.0) and otherwise use public `client.request()`. They never touch private attributes. Those fallbacks stay until the marketplace pull request removes them. +- The SDK sends the token as a `token` query parameter to api, kg, and nl.diffbot.com, and as `Authorization: Bearer` to llm.diffbot.com (Web Search). Errors are mapped to plugin-safe text and the token is redacted from them. + +### Credential fields + +From `provider/diffbot.yaml` `credentials_for_provider`: + +| Field | Type | Required | Default | Purpose | +| --- | --- | --- | --- | --- | +| `diffbot_token` | secret-input | yes | none | Diffbot API token. Format-checked (16 to 256 characters of `A-Za-z0-9._~+-`) and verified with the Account API when the credential is saved. | +| `request_timeout` | text-input | no | `30` | Seconds per Diffbot call, 5 to 300. The 300 ceiling is dify_plugin's request limit. | + +### Network domains + +`manifest.yaml` `network.domains`, unchanged from v0.1: + +- `api.diffbot.com` (Extract, Account, Crawl) +- `kg.diffbot.com` (Knowledge Graph DQL) +- `llm.diffbot.com` (Web Search) +- `nl.diffbot.com` (Natural Language) + +The plugin process only talks to these four hosts. Neither credentials nor parameters can change the SDK base URLs. User-supplied URLs (extract `url`, crawl `seeds`, crawl `api_url`) go to Diffbot as parameters, and Diffbot's own infrastructure fetches them. The plugin does not. ## Package the file they will attach -From a checkout of the plugin, with the Dify plugin CLI (this run used `dify-plugin-linux-amd64` 0.6.15): +From a checkout of the plugin, after `diffbot==3.4.0` is pinned, with the Dify plugin CLI (the v0.1 run used `dify-plugin-linux-amd64` 0.6.15; record the version you use): ```bash dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg @@ -42,7 +105,7 @@ dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report ``` -The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below already contains the summary for the current package. +The validator lives in [langgenius/dify-marketplace-toolkit](https://github.com/langgenius/dify-marketplace-toolkit). Exit code 0 means no blocking package errors. Warnings still need an explanation in the PR. The PR body below has `TODO(sdk-release)` placeholders where the new run's summary goes. `dify plugin package` produces an unsigned archive. Dify signs marketplace packages after merge. A third-party signature is only for Community Edition installs outside the marketplace: @@ -89,7 +152,7 @@ Later versions add a new file such as `diffbot/diffbot/diffbot-0.2.0.difypkg` an ## PR body to paste -Fill this from the live template. Do not delete sections. The risk box must be exactly one level. None or several becomes the `risk: missing` label. +Fill this in from the live template. Keep every section. Check exactly one risk box: none, or more than one, gets the `risk: missing` label. Do not paste it as-is. Every `TODO(sdk-release)` line has to be replaced with real results first. ````markdown # Plugin Submission @@ -109,15 +172,15 @@ Fill this from the live template. Do not delete sections. The risk box must be e ## What changed -New tool plugin for Diffbot. Seven tools: +New tool plugin for Diffbot, built on the official Diffbot Python SDK (`diffbot==3.4.0`). Eight tools: - Web search on llm.diffbot.com -- Extract page on api.diffbot.com (auto or a specific page type) +- Extract a page on api.diffbot.com (auto or a specific page type) - Knowledge Graph search for organizations, people, and news -- Enhance entity for one organization or person +- Start a Diffbot crawl job, and list, check, fetch results of, or delete crawl jobs (delete is destructive) - Resolve entities in pasted text to Knowledge Graph ids -Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). Saving the credential calls the Account API. The plugin does not crawl, run raw DQL, or accept email, phone, or IP as inputs. +Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). The plugin calls the Account API to check the token when the credential is saved. One optional credential field sets the request timeout (5 to 300 seconds). The private-host guard is always on. The plugin has no raw DQL tool and takes no email, phone, or IP inputs. ## Risk level @@ -125,7 +188,7 @@ Authentication is a Diffbot token (free at https://app.diffbot.com/get-started). - [ ] Medium risk - [x] High risk -Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. People search, Enhance, and entity resolution handle personal data from public profiles and from text the user pastes (names, employers, locations, LinkedIn URLs). The submission rules say to pick the higher level when more than one could apply. Search-only peers have been accepted as low risk; this plugin does more than search, so it is marked high. +Extract and crawl both send user-supplied http(s) URLs to Diffbot, and Diffbot fetches them. A crawl follows links from up to 20 seeds, up to 10,000 pages, and keeps running in the background on the user's Diffbot account after the tool call returns. That falls under "proxy or crawl arbitrary URLs" in the high-risk definition. The crawl jobs tool can permanently delete a crawl job and its data. People search and entity resolution handle personal data from public profiles and from text the user pastes. ## Required checks @@ -139,41 +202,50 @@ Extract sends a user-supplied http(s) URL to Diffbot, which fetches that URL. Pe - [x] The plugin includes `PRIVACY.md` or a hosted privacy policy, and `manifest.yaml` references it. - [x] All user-facing text is primarily in English, with any localized README files following the [i18n guidance](https://docs.dify.ai/en/develop-plugin/features-and-specs/plugin-types/multilingual-readme). + + ## Security and privacy notes -- Input constraints and security boundaries: outbound hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The user cannot pass a raw URL for the plugin to request, and cannot pass raw DQL. Extract rejects non-http(s) URLs, URLs with userinfo, and literal local, private, or link-local hosts. Timeouts are 15 to 45 seconds. Result counts are capped (10 search rows, 3 enhance matches, 20,000 characters of text). Errors are rewritten so the token is not returned. Email and phone fields are dropped from Knowledge Graph records before they reach the model. -- Arbitrary URL fetching: Extract asks Diffbot to fetch one user-supplied public http(s) URL. The plugin itself only calls the four Diffbot hosts above. Scheme, userinfo, and literal private-host checks are described above. -- Personal data: People search, Enhance, and entity resolution process names, public profile fields, and location text. They do not accept email, phone, or IP input. Outputs drop email and phone if the API includes them. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could be returned; the privacy policy tells users not to send data about children. -- Financial data: Organization search can return public revenue and headcount figures. It does not move money. +- Outbound hosts: the plugin process only calls api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com, through the Diffbot SDK over httpx. No credential or parameter changes those base URLs. A guarded httpx transport refuses non-http(s) schemes and private, loopback, link-local, and metadata hosts before connecting. +- User-supplied URLs: extract `url`, crawl `seeds`, and crawl `api_url` must be http(s), have no userinfo, be at most 2,048 characters, and not be a local name or a literal private, loopback, link-local, reserved, or metadata address. Bare domains get `https://`. The check is on the literal host only. The plugin does not resolve DNS, because Diffbot does the fetching from its own infrastructure. +- Crawling: crawl jobs run on Diffbot in the background and use the account's quota until they finish or are deleted. Limits: 20 seeds, 10,000 pages crawled and processed, 10 hops. The crawl stays on the seed domains by default. `obey_robots` is off in the form by default. When it is off the SDK leaves out `obeyRobots`, so the Diffbot crawler's own default applies. A custom request header can be set and is sent to the crawled sites, and the parameter text tells users not to put secrets there. Job names are at most 30 characters. +- Destructive action: `crawl_jobs` with `delete` permanently removes a crawl job and its collected data. The action is labeled destructive, and the model is told to delete only on an explicit user request. List, status, results, and delete cover every crawl job on the token's account, not only jobs started from Dify. +- Limits: each call has a configurable timeout of 5 to 300 seconds (default 30). Web search returns at most 50 results, and Knowledge Graph searches return at most 100 records. Crawl results return at most 1,000 rows with cells clipped to 500 characters. Entity resolution accepts up to 20,000 characters. Long text fields are clipped before they reach the model. Errors are rewritten so they never contain the token. Email and phone fields are removed from Knowledge Graph records. +- Personal data: people search and entity resolution process names, public profile fields, and location text. Neither tool accepts email, phone, or IP input. The Knowledge Graph cannot be filtered by age, so a public profile of a minor could come back. The privacy policy tells users not to send data about children. +- Financial data: organization search can return public revenue and headcount figures. The plugin does not move money. - No command execution, code execution, shell, SQL against a customer database, SSH/SFTP, file operations, or browser automation. -- Queries, URLs, and pasted text are sent to Diffbot over HTTPS. The plugin does not store them. See PRIVACY.md and https://www.diffbot.com/company/privacy. +- Queries, URLs, crawl settings, and pasted text go to Diffbot over HTTPS. The plugin stores none of it. Crawl jobs and their results are kept in the user's Diffbot account until deleted. See PRIVACY.md and https://www.diffbot.com/company/privacy. ## Local validation ```text -dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI 0.6.15 +dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg # dify CLI dify plugin checksum dist/diffbot-0.1.0.difypkg python3 validator/validate-difypkg.py dist/diffbot-0.1.0.difypkg --output-dir ./validation-report --pr-body-file pr-body.md ``` -Checksum `6abed6eb542446e656d598ed5bb98476cc2ec971a3254a7f295661ce10ca84a8`. +Checksum: TODO(sdk-release): rerun marketplace validator. -Validator: exit code 0, blocking failures 0. package_contents, package_secrets, package_binaries, manifest_metadata, readme_metadata, package_dependencies, python_compile, sensitive_capabilities: PASS. python_safety, prohibited_financial_activity: PASS. access_domains: static scan found only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, nl.diffbot.com, matching `network.domains`. dependency_vulnerabilities: `dify_plugin==0.11.0` and `requests==2.34.2` pinned, no known vulnerabilities. +Validator: TODO(sdk-release): rerun marketplace validator. Report exit code, blocking failures, and each check, especially: +- dependency_vulnerabilities: expected pins are `dify_plugin==0.11.0` and `diffbot==3.4.0`. Record what OSV reports, including transitive `httpx`. +- access_domains: expected hosts are only api.diffbot.com, kg.diffbot.com, llm.diffbot.com, and nl.diffbot.com. The base URLs now sit inside the SDK, so record what the static scan actually finds. +- python_safety: record the result for the new SDK/httpx code (`tools/sdk.py`, crawl tools). -Unit tests: `pytest` 31 passed (mocked HTTP). Plugin start-up: `validator/test-plugin-install.py` with `INSTALL_METHOD=serverless` succeeded. +Unit tests: `pytest` 109 passed (mocked HTTP). Plugin start-up (`validator/test-plugin-install.py`, `INSTALL_METHOD=serverless`): TODO(sdk-release): rerun marketplace validator. -Live smoke test against the Diffbot API on 10 November 2026, calling each tool class directly with a real token: credential check, web search, extract, organization search, people search, news search (newest first), enhance, and entity resolution all returned results. +Live smoke test: TODO(sdk-release): rerun every tool, including a crawl start/status/results/delete cycle, against the Diffbot API with a real token. ## Reviewer notes -- Not yet installed on Dify Community Edition or Dify Cloud. The tools were exercised live outside Dify, and the package passes the toolkit's start-up test. Results from a Community Edition and Dify Cloud install will be added to this PR before it is marked ready for review. -- `meta.minimum_dify_version` is 1.14.0, from the dify_plugin 0.11.0 compatibility table (SDK 0.8.1 and later). The plugin uses no later gated feature. +- Not yet installed on Dify Community Edition or Dify Cloud. Results from both will be added to this PR before it is marked ready for review. +- `meta.minimum_dify_version` is 1.14.0, taken from the dify_plugin 0.11.0 compatibility table (SDK 0.8.1 and later). The plugin uses no later gated feature. +- Crawl is asynchronous on purpose. A Dify tool call is capped at 300 seconds, so `crawl_start` returns a job name and the agent polls with `crawl_jobs`. - Industry filter on organization search is an exact `categories.name`. Free text is not resolved through the ontology in this version. -- Job titles match profile text as written. `CEO` and `Chief Executive Officer` may return different people; the tool description tells the model to try both. +- Job titles match profile text as written. `CEO` and `Chief Executive Officer` may return different people. The tool description tells the model to try both. - The icon is Diffbot's own 2026 logomark, published by Diffbot. ```` -Leave the "tested on Community Edition and Dify Cloud" box unchecked until that test exists. Do not check a box the package does not meet. Explain the gap under reviewer notes instead. +Leave the "tested on Community Edition and Dify Cloud" box unchecked until that test has been done. Never check a box the package does not meet. Explain the gap under reviewer notes instead. ## What reviewers look for @@ -217,79 +289,58 @@ No Diffbot Dify plugin showed up in the Marketplace search, in the Diffbot GitHu A third-party write-up of Marketplace tool installs on 16 October 2026 put Tavily, Jina, and Firecrawl in the same band (about 399k, 227k, and 204k). The 9 November API numbers above are higher and are the ones to cite. + ## v1 tool choice -`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. The Enhance HTTP API is still documented at `https://kg.diffbot.com/kg/v3/enhance`. +`diffbot-mcp` tools on 9 November 2026: `extract`, `search_web`, `crawl`, `resolve_entities`, `search_news`, `search_organizations`, `search_people`, `search_places`, `search_deals`, `dql`, `dql_probe`, `dql_ontology`. The October 2026 changelog removed MCP `enhance` in favor of `resolve_entities` plus the typed searches. This plugin dropped its `enhance_entity` tool for the same reason (commit 7323b7b), so it matches the Diffbot Python SDK surface. -v1 keeps the tools a Dify user already expects from Tavily or Firecrawl (search and extract) and adds the Knowledge Graph lookups that distinguish Diffbot (organizations, people, news, enhance, entity linking). It does not expose raw DQL, so a model cannot send an unbounded query. It does not start crawls, which are background jobs and a clearer "crawl arbitrary URLs" case. Places, deals, ontology browse, and hit-count probes can be a later version once v1 has been installed for real. +v1 has what a Dify user expects from Tavily or Firecrawl: search, extract, and crawl. It adds the Knowledge Graph lookups that set Diffbot apart: organizations, people, news, and entity linking. There is no raw DQL tool, so a model cannot send an unbounded query. Crawl is in v1 because Tavily and Firecrawl both offer it. That decision moves the plugin into the "crawl arbitrary URLs" high-risk case. Places, deals, ontology browse, and hit-count probes can come in a later version once v1 has been installed for real. -Authentication matches the docs, with one exception. Extract, Account, DQL, Enhance, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` is documented with `Authorization: Bearer`, and the plugin does that so the token is not written into the query string for that host. +The SDK handles authentication as the API docs describe. Extract, Account, Crawl, DQL, and Natural Language take `token` as a query parameter on `api.diffbot.com`, `kg.diffbot.com`, and `nl.diffbot.com`. Web Search on `llm.diffbot.com` uses `Authorization: Bearer`, so the token does not appear in that host's query string. ## Results for the current package -Unit tests, no network: +Unit tests, no network, 17 November 2026, against a local editable checkout of the unreleased SDK: ```text pytest -31 passed -``` - -Dify CLI 0.6.15: - -```text -dify plugin package ./diffbot -o dist/diffbot-0.1.0.difypkg -dify plugin checksum dist/diffbot-0.1.0.difypkg -# checksum 6abed6eb542446e656d598ed5bb98476cc2ec971a3254a7f295661ce10ca84a8 +109 passed ``` -The archive is about 62 KB, 29 files: the manifest, provider, seven tool pairs, client code, README, PRIVACY.md, icon, `main.py`, `requirements.txt`, and `pyproject.toml`. `.env.example` is kept in git for remote debugging and excluded from the package by `.difyignore`. `dist/` is not tracked in git. +Package, checksum, archive size and file list: **TODO(sdk-release): rerun marketplace validator.** The package should hold the manifest, provider, eight tool YAML/Python pairs, the shared modules (`client.py`, `common.py`, `sdk.py`, `sdk_types.py`, `dql_builder.py`, `shape.py`), README, PRIVACY.md, icon, `main.py`, `requirements.txt`, and `pyproject.toml`. Check the real archive against that list. `.env.example` stays in git for remote debugging and `.difyignore` keeps it out of the package. `dist/` is not tracked in git. -Marketplace toolkit validator, with `--pr-body-file` and the OSV lookup: +Marketplace toolkit validator: **TODO(sdk-release): rerun marketplace validator.** The pre-SDK v0.1 numbers that used to be here (31 tests, checksum `6abed6eb…`, `requests==2.34.2` with no known vulnerabilities, a four-host access_domains scan of the plugin's own code, a python_safety pass) were measured on a different dependency set and code path. Do not reuse them. Record dependency_vulnerabilities, access_domains, python_safety, the PR-body checks, and the serverless start-up test from the new run. -- Exit code 0. Blocking failures: 0. -- package_contents, package_secrets, package_binaries, manifest_metadata, readme_metadata, package_dependencies, python_compile, and sensitive_capabilities passed. python_safety and prohibited_financial_activity passed. -- access_domains: the static scan found only `api.diffbot.com`, `kg.diffbot.com`, `llm.diffbot.com`, and `nl.diffbot.com`, which are the `network.domains` entries. -- dependency_vulnerabilities: pinned `dify_plugin==0.11.0` and `requests==2.34.2` had no known vulnerabilities. -- `check-pr-body.py` and `check-pr-review-notes.py` passed on the PR body. -- `test-plugin-install.py` (`INSTALL_METHOD=serverless`) started the plugin and got a response on port 8080. - -Live smoke test, 10 November 2026, each tool class called directly with a real token (not inside Dify): - -```text -[PASS] credential check (2.8s): ok -[PASS] search_web (1.9s): text=Diffbot web search returned 3 result(s). 1. Knowledge Graph | Diffbot (score 0.992) https://www.diffbot.com/products/knowledge-graph/ # Knowledge Graph [](https://www.diffbot.com/products/knowl | json keys=['query', 'timeMs', 'results'] n=3 | first={"title": "Knowledge Graph | Diffbot", "date": null, "pageUrl": "https://www.diffbot.com/products/knowledge-graph/", "score": 0.992} -[PASS] extract (1.9s): text=Diffbot extract for https://www.diffbot.com/company/ About Diffbot | Diffbot (other) https://www.diffbot.com/company/about | json keys=['objects', 'humanLanguage', 'url', 'page_type'] n=1 | first={"title": "About Diffbot | Diffbot", "pageUrl": "https://www.diffbot.com/company/about", "type": "other"} -[PASS] search_organizations (1.5s): text=Organization search returned 2 record(s) of 4. 1. Diffbot — Menlo Park, United States diffbot.com American machine learning and knowledge management company 2. ROS Mobile Robots — Feldkirch, Aus | json keys=['dql', 'hits', 'returned', 'results'] n=2 | first={"name": "Diffbot", "diffbotUri": "http://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ"} -[PASS] search_people (2.4s): employer "Diffbot" + title "CEO" -> 1 public profile (Diffbot's founder and CEO); grouped employments.{...} query -[PASS] search_news (2.1s): text=News search returned 3 article(s) of 3552, newest publication date first. 1. Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein (WinFuture.de — d2026-11-10T19:07) https://winfut | json keys=['dql', 'hits', 'returned', 'results'] n=3 | first={"title": "Windows-Sicherheit: Microsoft stuft Family-Safety-App als veraltet ein", "date": "d2026-11-10T19:07", "pageUrl": "https://winfuture.de/news,162580.html"} -[PASS] enhance_entity (2.0s): text=Enhance returned 1 organization match(es). 1. Diffbot (score 0.943500057890868) diffbot.com http://diffbot.com/entity/EYX1i02YVPsuT7fPLUYgRhQ | json keys=['hits', 'matches'] n=1 | first={"score": 0.943500057890868} -[PASS] resolve_entities (1.8s): text=Resolved 4 entities from the supplied text. - Mike Tung [person] id=ESGMaGV9uP0SuTmfPTtNEoA - Menlo Park [location, administrative area, city] id=EQiAUr370NpW1gvx-V4fTrQ - Diffbot [organization] id=EY | json keys=['sentiment', 'entities', 'dql_filter'] n=4 -``` - -A live DQL check decided news ordering. `type:Article tags.label:"Microsoft" date>="2026-01-01"` with `sortBy:date` returned articles from 10 November 2026 first. With `revSortBy:date` it returned 1 January 2026 first. The plugin now uses `sortBy:date`. - -People search with employer `Diffbot` and title `Chief Executive Officer` returned an executive assistant at another company, because separate `employments.*` clauses could match different jobs. The plugin now groups them as `employments.{employer.name:"..." title:"..."}`. Titles still match profile text as written: `CEO` finds Diffbot's founder and `Chief Executive Officer` does not. +Live smoke test: **TODO(sdk-release): rerun.** The 10 November 2026 run covered the pre-SDK package (credential check, web search, extract, organization, people, and news search, and entity resolution). It did not cover crawl. Two behaviors it settled are still in the code. News uses `sortBy:date` (newest first). People search groups employer and title as `employments.{employer.name:"..." title:"..."}` so both clauses must match the same job. Recheck both in the new run. ## Open questions and risks Blockers before a real PR: +- `diffbot` 3.4.0 must be on PyPI and pinned, and the validator rerun on the resulting package. Until then the dependency, access-domain, and safety results are unknown. +- PRIVACY.md does not describe crawl yet. - The package has not been installed in Dify Community Edition or Dify Cloud. Do that, or write the limitation in the PR. - Whoever opens the PR must be allowed to publish under the `diffbot` author name. Open it from a fork owned by, or on behalf of, the `diffbot` organization. +Crawl scope and risk: + +- Crawling user-supplied URLs is the main reason this plugin is high risk. An agent can start a crawl of any public site the user or the model names. The crawl follows links up to the configured limits and runs on Diffbot's infrastructure, against the user's own Diffbot account and quota. The plugin's URL guard stops literal private and metadata targets. It does not resolve hostnames, and it cannot judge whether the user is allowed to crawl a site. That depends on Diffbot's terms and the user's account. +- The `obey_robots` form field defaults to off. The SDK sends `obeyRobots=1` only when it is on. When it is off, nothing is sent and the Diffbot crawler's own robots.txt default applies, whatever that default is. So the toggle can say "off" while the crawler is still obeying robots.txt, or the reverse. Confirm the API default before release and make the label match it. A reviewer may also want robots.txt respected by default in a Dify-facing tool. That is a product call, and the change is a one-line YAML edit. +- Crawl jobs outlive the tool call. A user who starts a large crawl from an agent keeps using quota after the conversation ends, and the plugin has no cleanup. The tool text says the job runs in the background, and `crawl_jobs` can stop it with `delete`. +- `delete` is irreversible and has no confirmation step in the plugin. All that keeps a model from deleting is the LLM description ("only use delete when the user explicitly asks"). It also reaches every crawl job on the token's account, including jobs created outside Dify. If reviewers object, the options are to split delete into its own tool that users can leave disabled, or to limit delete to job names the plugin generated. +- The custom header parameter is sent to every crawled site. The parameter text warns against putting secrets there, but the plugin does not enforce that. + Product and API risks: -- Free tokens may not include Web Search, Knowledge Graph, Enhance, and Natural Language together. The Account API can succeed while a later call returns 401 or 403. The 10 November live test used one Diffbot token with all of them enabled; a free-plan token has not been tried. -- Organization industry is an exact category string. The MCP server resolves free text through the ontology. v1 will return empty results for "software" when the category is "Software Companies". Either document that sharply or add ontology resolution in a later version. -- Job-title abbreviations are not expanded (`CEO` versus `Chief Executive Officer`). The MCP server does expand them. The tool description tells the model to try both forms, but people search can still under-count until expansion exists. -- Web Search might also accept `token` as a query parameter. This plugin uses only the documented bearer header. If a token type rejects the header, the tool fails closed. That is a live test, not a guess to code around. -- Enhance can spend more credits when `size` is above 1. The cap is 3. `refresh` and `search=true` are not exposed because they recrawl or do an extra web merge and are slow. Say that in support docs so customers do not expect them. +- Free tokens may not include Web Search, Knowledge Graph, Natural Language, and Crawl together. The Account API check can pass while a later call returns 401 or 403. No free-plan token has been tested. +- Organization industry is an exact category string. The MCP server resolves free text through the ontology. v1 returns nothing for "software" when the category is "Software Companies". Either document that clearly or add ontology resolution in a later version. +- Job-title abbreviations are not expanded (`CEO` versus `Chief Executive Officer`). The MCP server does expand them. The tool description tells the model to try both forms, but people search can still miss matches until expansion exists. Review and policy risks: -- High risk is the honest box. A reviewer can still ask for a narrower tool set. Dropping Extract would remove user-supplied URL fetch, but people search would still be personal data. Dropping people, Enhance, and entity resolution would look more like Tavily and might be argued as medium, because the remaining URL fetch is still "user-provided URLs" in the medium definition and "proxy or crawl arbitrary URLs" in the high definition. The written rule is to choose the higher label. Do not mark this low because Exa did. Exa's PR text said it does not fetch arbitrary URLs. This plugin does ask Diffbot to fetch them. -- Diffbot's privacy policy says the service is not directed at anyone 18 or under, and gives parents a contact address for privacy requests. The Knowledge Graph can still return a public profile of a younger person, and this plugin cannot filter by age. Call that out in review rather than claiming the plugin is age-safe. -- Dify Cloud customers will send workspace prompts to Diffbot. Whether that needs a DPA or a customer-facing subprocessors note is a Diffbot legal question. The plugin policy links Diffbot's policy and says the plugin itself stores nothing. It does not answer the Dify customer's contract question. +- High risk is the honest box. Extract and crawl both have Diffbot fetch user-supplied URLs, and crawl is the plain "crawl arbitrary URLs" case. A reviewer can still ask for a narrower tool set. Dropping crawl would leave single-page Extract ("user-provided URLs") and people search (personal data), which is still at least medium. The written rule is to pick the higher label. Do not mark this low because Exa did: Exa's PR said it does not fetch arbitrary URLs, and this plugin does. +- Diffbot's privacy policy says the service is not directed at anyone 18 or under, and gives parents a contact address for privacy requests. The Knowledge Graph can still return a public profile of a younger person, and this plugin cannot filter by age. Raise that in review rather than claiming the plugin is age-safe. +- Dify Cloud customers will send workspace prompts and crawl targets to Diffbot. Whether that needs a DPA or a customer-facing subprocessors note is a question for Diffbot legal. - A merged PR might not produce a listing. Watch the Marketplace after merge, as Exa had to. - `meta.minimum_dify_version` is 1.14.0, taken from the SDK compatibility table rather than an install on 1.14.0. If the Community Edition test runs on a newer version and a reviewer asks, either test 1.14.0 or raise the floor to the tested version. diff --git a/diffbot/PRIVACY.md b/diffbot/PRIVACY.md index f2ebafd..9a98ee5 100644 --- a/diffbot/PRIVACY.md +++ b/diffbot/PRIVACY.md @@ -18,10 +18,9 @@ Every tool call leaves the Dify workspace and is processed by Diffbot Technologi | Search query, result count, snippet size | `llm.diffbot.com` Web Search | Run a web search. The token is sent as a bearer header. | | Page URL and page type | `api.diffbot.com` Extract | Fetch and structure one public page. Diffbot retrieves the URL. | | Names, homepages, categories, places, employee and revenue bounds, article phrases, publishers, dates | `kg.diffbot.com` DQL | Search the Knowledge Graph. | -| Person or organization name, public URL, location text, employer, title | `kg.diffbot.com` Enhance | Match a Knowledge Graph record. | | Plain text from the user, up to 20,000 characters, and an optional language code | `nl.diffbot.com` | Recognize and link named entities. | -The plugin does not offer parameters for email address, phone number, or IP address, including on Enhance, where the Diffbot API supports those fields. If a Knowledge Graph response still contains email or phone values, the plugin drops them before returning the tool result. It also drops article HTML and, unless the news tool is asked for an excerpt, article body text. +The plugin does not offer parameters for email address, phone number, or IP address. If a Knowledge Graph response still contains email or phone values, the plugin drops them before returning the tool result. It also drops article HTML and, unless the news tool is asked for an excerpt, article body text. People and organization results can still contain personal data that is already public on the web: names, job titles, employers, cities, countries, homepage URLs, and LinkedIn profile URLs. Diffbot describes people in its index as search subjects whose information was published on publicly available sites. News results can quote public articles. Entity resolution can identify people named in text the user pasted. diff --git a/diffbot/README.md b/diffbot/README.md index f777a66..61b7dd2 100644 --- a/diffbot/README.md +++ b/diffbot/README.md @@ -2,7 +2,7 @@ Search the public web and the Diffbot Knowledge Graph from a Dify agent or workflow. -Diffbot structures pages and keeps a knowledge graph of organizations, people, and articles. This plugin exposes a small v1 of that API: web search, page extraction, organization and people lookup, news search, entity enhancement, and named-entity resolution. A free token from [app.diffbot.com/get-started](https://app.diffbot.com/get-started) is enough to authenticate. +Diffbot structures pages and keeps a knowledge graph of organizations, people, and articles. This plugin exposes a small v1 of that API: web search, page extraction, organization and people lookup, news search, and named-entity resolution. A free token from [app.diffbot.com/get-started](https://app.diffbot.com/get-started) is enough to authenticate. Published by Diffbot. Source: https://github.com/diffbot/dify-diffbot. Support: support@diffbot.com. @@ -12,29 +12,31 @@ Published by Diffbot. Source: https://github.com/diffbot/dify-diffbot. Support: | --- | --- | --- | | Web search | `GET https://llm.diffbot.com/api/v1/web_search` | Ranked pages with a score, URL, date, and passage. A query that is only `url:https://...` reads the indexed copy. | | Extract page | `GET https://api.diffbot.com/v3/analyze` or a typed extract path | Live extraction of one public page. `article` is the usual choice for news and blogs. | -| Search organizations | `GET https://kg.diffbot.com/kg/v3/dql` | Companies by name, homepage, category, headquarters, headcount, revenue, or public status. | +| Search organizations | `POST https://kg.diffbot.com/kg/v3/dql` | Companies by name, homepage, category, headquarters, headcount, revenue, or public status. | | Search people | same DQL endpoint | Public professional profiles by name, employer, title, or location. | | Search news | same DQL endpoint | Articles by mentioned entity, phrase, publisher, language, or date. | -| Enhance entity | `GET https://kg.diffbot.com/kg/v3/enhance` | Best Knowledge Graph match for a person or organization from a name, URL, or employer. | +| Start crawl | `GET https://api.diffbot.com/v3/crawl` | Start a background crawl of public seed URLs. Returns a job name, never results. | +| Crawl jobs | `/v3/crawl`, `/v3/crawl/data` | List jobs, check status, fetch results (up to 1000 rows, truncated), or delete a job. **Delete is destructive** and removes the job and its data. | | Resolve entities | `POST https://nl.diffbot.com/v1/` | Names in a passage, linked to Knowledge Graph ids when Diffbot can match them. | -Crawl, raw DQL, ontology lookup, places, and deals are not in this version. Industry filters must already be exact category names such as `Software Companies`. The plugin does not resolve free text onto the taxonomy. +Raw DQL, ontology lookup, Ask, places, and deals are not in this version. Industry filters must already be exact category names such as `Software Companies`. The plugin does not resolve free text onto the taxonomy. ## Credentials 1. Create a token at [app.diffbot.com/get-started](https://app.diffbot.com/get-started). 2. In Dify, open Plugins, install this package, and choose Diffbot. 3. Paste the token into **Diffbot token**. +4. Optional: set **Request timeout** (5 to 300 seconds, default 30). Dify stores the value as a secret. The plugin sends it only to the Diffbot hosts listed in `manifest.yaml`. Web Search uses `Authorization: Bearer`. The other calls use the `token` query parameter, which is how those APIs authenticate. Saving the credential calls `GET https://api.diffbot.com/v4/account` with `days=1` and does not show the account payload. -The token needs access to Extract, Web Search, Knowledge Graph, Enhance, and Natural Language. A free plan may not include every product. If a call returns an authorization error after the account check succeeds, confirm that product is enabled for the token. +The token needs access to Extract, Web Search, Knowledge Graph, and Natural Language. A free plan may not include every product. If a call returns an authorization error after the account check succeeds, confirm that product is enabled for the token. ## Use the tools -In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits are small on purpose: at most 10 search rows, 3 enhance matches, and 20,000 characters of text for entity resolution. +In an Agent, add the Diffbot tools and ask for a company, a news topic, or a URL. In a Workflow, add a tool node and pick one action. Required fields are marked on each tool. Limits: up to 50 web results, 100 Knowledge Graph rows per call (page with Offset), and 20,000 characters of text for entity resolution. Extract has optional caller-set options (fields, timeout, mode, paging, natural language, and others). -Extract accepts only `http` and `https` URLs. Localhost, link-local, and private literal addresses are rejected before the request is sent. +Extract accepts `http` and `https` URLs; a bare domain such as `example.com` gets `https://`. Localhost, link-local, and private literal addresses are rejected before the request is sent. ## Install the package locally diff --git a/diffbot/provider/diffbot.py b/diffbot/provider/diffbot.py index 769c987..b4e6d98 100644 --- a/diffbot/provider/diffbot.py +++ b/diffbot/provider/diffbot.py @@ -6,13 +6,18 @@ from dify_plugin.errors.tool import ToolProviderCredentialValidationError from tools.client import DiffbotError -from tools.common import make_client +from tools.sdk import make_client, mapped_errors, verify_token class DiffbotProvider(ToolProvider): def _validate_credentials(self, credentials: dict[str, Any]) -> None: try: - make_client(credentials).validate_token() + client = make_client(credentials) + with client, mapped_errors(client): + # verify_token() never hands back the account payload (it contains the token). + valid = verify_token(client) + if not valid: + raise DiffbotError("Invalid or unauthorized Diffbot token.") except DiffbotError as exc: raise ToolProviderCredentialValidationError(str(exc)) from exc except Exception as exc: diff --git a/diffbot/provider/diffbot.yaml b/diffbot/provider/diffbot.yaml index 8ddce53..297d650 100644 --- a/diffbot/provider/diffbot.yaml +++ b/diffbot/provider/diffbot.yaml @@ -21,13 +21,24 @@ credentials_for_provider: help: en_US: A free token is available at app.diffbot.com/get-started. Dify stores it as a secret and the plugin sends it only to Diffbot. url: https://app.diffbot.com/get-started + request_timeout: + type: text-input + required: false + default: "30" + label: + en_US: Request timeout (seconds) + placeholder: + en_US: "30" + help: + en_US: Seconds to wait for each Diffbot call, from 5 to 300. Leave blank for 30. tools: - tools/search_web.yaml - tools/extract.yaml - tools/search_organizations.yaml - tools/search_people.yaml - tools/search_news.yaml - - tools/enhance_entity.yaml + - tools/crawl_start.yaml + - tools/crawl_jobs.yaml - tools/resolve_entities.yaml extra: python: diff --git a/diffbot/pyproject.toml b/diffbot/pyproject.toml index e26a969..991a3d9 100644 --- a/diffbot/pyproject.toml +++ b/diffbot/pyproject.toml @@ -6,5 +6,5 @@ readme = "README.md" requires-python = ">=3.12" dependencies = [ "dify_plugin==0.11.0", - "requests==2.34.2", + "diffbot==3.4.0", ] diff --git a/diffbot/requirements.txt b/diffbot/requirements.txt index c0ee9fe..413e758 100644 --- a/diffbot/requirements.txt +++ b/diffbot/requirements.txt @@ -1,2 +1,2 @@ dify_plugin==0.11.0 -requests==2.34.2 +diffbot==3.4.0 diff --git a/diffbot/tools/client.py b/diffbot/tools/client.py index d157272..6c5d13c 100644 --- a/diffbot/tools/client.py +++ b/diffbot/tools/client.py @@ -1,27 +1,15 @@ -"""HTTP client for the Diffbot APIs used by this plugin. +"""Plugin policy shared by every tool: token format, URL/host guard, safe errors. -Authentication is a Diffbot token. Extract, Knowledge Graph, Enhance, Account, -and Natural Language calls send it as the ``token`` query parameter. Web Search -on ``llm.diffbot.com`` uses ``Authorization: Bearer`` instead, which is what -the current Web Search reference documents. +HTTP itself is done by the Diffbot Python SDK (see ``tools/sdk.py``); this module +holds only the rules the plugin enforces on top of it. """ from __future__ import annotations import ipaddress import re -from typing import Any from urllib.parse import urlparse -import requests - -ACCOUNT_URL = "https://api.diffbot.com/v4/account" -EXTRACT_BASE = "https://api.diffbot.com/v3" -WEB_SEARCH_URL = "https://llm.diffbot.com/api/v1/web_search" -DQL_URL = "https://kg.diffbot.com/kg/v3/dql" -ENHANCE_URL = "https://kg.diffbot.com/kg/v3/enhance" -NLP_URL = "https://nl.diffbot.com/v1/" - TOKEN_PATTERN = re.compile(r"^[A-Za-z0-9._~+\-]{16,256}$") BLOCKED_HOST_SUFFIXES = (".local", ".localhost", ".internal") BLOCKED_HOSTS = { @@ -31,186 +19,16 @@ "metadata.google", } -TIMEOUTS = { - "account": 15, - "search": 30, - "extract": 45, - "dql": 30, - "enhance": 30, - "nlp": 30, -} - class DiffbotError(Exception): """A safe, user-facing Diffbot failure. Messages never include the token.""" -class DiffbotClient: - def __init__(self, token: str, session: requests.Session | None = None) -> None: - self.token = normalize_token(token) - self.session = session or requests.Session() - - def validate_token(self) -> None: - """Confirm the token with the Account API without returning account details. - - ``days=1`` keeps the usage history small. The response body contains the - token itself, so this method returns nothing and callers must not log it. - """ - data = self._request_json( - "GET", - ACCOUNT_URL, - params={"token": self.token, "days": 1}, - timeout=TIMEOUTS["account"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot account check returned an unexpected response.") - if not any(key in data for key in ("plan", "status", "email", "name")): - raise DiffbotError("Diffbot account check did not return account details.") - - def web_search( - self, - query: str, - *, - num_results: int, - snippet_length: str, - max_tokens: int | None, - rerank: str | None, - ) -> dict[str, Any]: - params: dict[str, Any] = { - "text": query, - "size": num_results, - "snippetLength": snippet_length.upper(), - } - if max_tokens is not None: - params["maxTokens"] = max_tokens - if rerank: - params["rerank"] = rerank - data = self._request_json( - "GET", - WEB_SEARCH_URL, - params=params, - headers={"Authorization": f"Bearer {self.token}", "Accept": "application/json"}, - timeout=TIMEOUTS["search"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot web search returned an unexpected response.") - return data - - def extract(self, url: str, *, page_type: str, markdown: bool) -> dict[str, Any]: - endpoint = "analyze" if page_type == "auto" else page_type - params: dict[str, Any] = { - "token": self.token, - "url": url, - "timeout": 30000, - } - if markdown: - params["mode"] = "llm" - data = self._request_json( - "GET", - f"{EXTRACT_BASE}/{endpoint}", - params=params, - timeout=TIMEOUTS["extract"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot extract returned an unexpected response.") - return data - - def dql(self, query: str, *, size: int) -> dict[str, Any]: - data = self._request_json( - "GET", - DQL_URL, - params={"token": self.token, "query": query, "size": size}, - timeout=TIMEOUTS["dql"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot Knowledge Graph search returned an unexpected response.") - return data - - def enhance(self, params: dict[str, Any]) -> dict[str, Any]: - data = self._request_json( - "GET", - ENHANCE_URL, - params={"token": self.token, **params}, - timeout=TIMEOUTS["enhance"], - ) - if not isinstance(data, dict): - raise DiffbotError("Diffbot Enhance returned an unexpected response.") - return data - - def resolve_entities(self, text: str, *, lang: str) -> dict[str, Any]: - data = self._request_json( - "POST", - NLP_URL, - params={"token": self.token, "fields": "entities,sentiment"}, - json_body=[{"lang": lang, "format": "plain text", "content": text}], - timeout=TIMEOUTS["nlp"], - ) - if isinstance(data, list): - data = data[0] if data else {} - if not isinstance(data, dict): - raise DiffbotError("Diffbot entity resolution returned an unexpected response.") - return data - - def _request_json( - self, - method: str, - url: str, - *, - params: dict[str, Any] | None = None, - json_body: Any = None, - headers: dict[str, str] | None = None, - timeout: float, - ) -> Any: - try: - response = self.session.request( - method, - url, - params=params, - json=json_body, - headers=headers, - timeout=timeout, - ) - except requests.Timeout as exc: - raise DiffbotError("The Diffbot request timed out.") from exc - except requests.RequestException as exc: - raise DiffbotError("The Diffbot request failed before a response was received.") from exc - - if response.status_code in (401, 403): - raise DiffbotError("Invalid or unauthorized Diffbot token.") - if response.status_code == 429: - raise DiffbotError("Diffbot rate limit exceeded. Retry later.") - if response.status_code >= 400: - raise DiffbotError( - f"Diffbot API error {response.status_code}: {self._safe_body_message(response)}" - ) - - try: - data = response.json() - except ValueError as exc: - raise DiffbotError("Diffbot returned a non-JSON response.") from exc - - if isinstance(data, dict) and data.get("errorCode"): - message = str(data.get("error") or data.get("message") or "request failed") - lowered = message.lower() - if "token" in lowered or "authorized" in lowered or "unauthor" in lowered: - raise DiffbotError("Invalid or unauthorized Diffbot token.") - raise DiffbotError(f"Diffbot request failed: {self._redact(message)}") - return data - - def _safe_body_message(self, response: requests.Response) -> str: - try: - data = response.json() - except ValueError: - return "request failed" - if isinstance(data, dict): - message = str(data.get("error") or data.get("message") or "request failed") - return self._redact(message)[:300] - return "request failed" - - def _redact(self, text: str) -> str: - if self.token and self.token in text: - text = text.replace(self.token, "[redacted]") - return text +def redact(text: str, token: str | None) -> str: + """Remove the token (and any ``token=`` query value) from text.""" + if token: + text = text.replace(token, "[redacted]") + return re.sub(r"(token=)[^&\s\"']+", r"\1[redacted]", text) def normalize_token(raw: str) -> str: @@ -244,7 +62,13 @@ def validate_public_http_url(url: str) -> str: raise DiffbotError("URL must start with http:// or https://.") if parsed.username or parsed.password: raise DiffbotError("URL must not include a username or password.") - host = (parsed.hostname or "").rstrip(".").lower() + check_public_host(parsed.hostname or "") + return url + + +def check_public_host(hostname: str) -> None: + """Reject empty, local-named, metadata, and private/loopback literal hosts.""" + host = hostname.rstrip(".").lower() if not host: raise DiffbotError("URL must include a host.") if host in BLOCKED_HOSTS or host.endswith(BLOCKED_HOST_SUFFIXES): @@ -252,8 +76,8 @@ def validate_public_http_url(url: str) -> str: try: ip = ipaddress.ip_address(host) except ValueError: - ip = None - if ip is not None and ( + return + if ( ip.is_private or ip.is_loopback or ip.is_link_local @@ -262,4 +86,13 @@ def validate_public_http_url(url: str) -> str: or ip.is_unspecified ): raise DiffbotError("URL host is not allowed.") - return url + + +def normalize_page_url(url: str) -> str: + """Prepend https:// to a bare domain (as the SDK does), then run the host guard.""" + if not isinstance(url, str): + raise DiffbotError("A page URL is required.") + url = url.strip() + if url and "://" not in url: + url = f"https://{url}" + return validate_public_http_url(url) diff --git a/diffbot/tools/common.py b/diffbot/tools/common.py index 8a3ae4b..a4aa702 100644 --- a/diffbot/tools/common.py +++ b/diffbot/tools/common.py @@ -7,19 +7,11 @@ from dify_plugin.entities.tool import ToolInvokeMessage -from tools.client import DiffbotClient +from tools.sdk import as_dict, make_client, mapped_errors # noqa: F401 (re-exported for tools) from tools.dql_builder import bounded_number from tools.shape import clip -def make_client(credentials: dict[str, Any] | None) -> DiffbotClient: - raw = "" - if credentials: - value = credentials.get("diffbot_token", "") - raw = value if isinstance(value, str) else "" - return DiffbotClient(raw) - - def optional_text(parameters: dict[str, Any], name: str) -> str: value = parameters.get(name) if value is None: @@ -35,6 +27,22 @@ def result_size(parameters: dict[str, Any], *, default: int, maximum: int) -> in return int(number) +def result_offset(parameters: dict[str, Any]) -> int: + value = parameters.get("offset") + if value is None or value == "": + return 0 + return int(bounded_number(value, label="offset", minimum=0, maximum=100000, integer=True)) + + +def run_dql(credentials: dict[str, Any] | None, query: str, *, size: int, offset: int) -> dict[str, Any]: + """Run a DQL query through the SDK (POST) and record the query that ran.""" + client = make_client(credentials) + with client, mapped_errors(client): + data = as_dict(client.dql(query, size=size, from_=offset), "Knowledge Graph search") + data["dql"] = query + return data + + def messages(tool: Any, summary: str, payload: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: yield tool.create_text_message(clip(summary)) yield tool.create_json_message(payload) diff --git a/diffbot/tools/crawl_jobs.py b/diffbot/tools/crawl_jobs.py new file mode 100644 index 0000000..7f552fb --- /dev/null +++ b/diffbot/tools/crawl_jobs.py @@ -0,0 +1,81 @@ +"""List, inspect, fetch results of, and delete Diffbot crawl jobs.""" + +from collections.abc import Generator +from typing import Any + +from dify_plugin import Tool +from dify_plugin.entities.tool import ToolInvokeMessage + +from tools.client import DiffbotError +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import crawl_download, crawl_get_job +from tools.shape import clip, summarize_lines + +ACTIONS = {"list", "status", "results", "delete"} +RESULT_LIMIT = 1000 +DEFAULT_RESULT_LIMIT = 100 # matches the yaml default +JOB_FIELDS = ("name", "jobStatus", "pageCrawlAttempts", "pageCrawlSuccesses", "pageProcessAttempts", + "pageProcessSuccesses", "objectsFound", "urlsFound", "maxToCrawl", "maxToProcess") + + +def _job_summary(job: dict[str, Any]) -> dict[str, Any]: + out = {key: job[key] for key in JOB_FIELDS if key in job} + status = out.get("jobStatus") + if isinstance(status, dict): + out["jobStatus"] = {k: status[k] for k in ("status", "message") if k in status} + return out + + +def _cell(value: Any) -> Any: + return clip(value, 500) if isinstance(value, str) else value + + +class CrawlJobsTool(Tool): + def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: + action = optional_text(tool_parameters, "action").lower() + if action not in ACTIONS: + raise DiffbotError("action must be list, status, results, or delete.") + job_name = optional_text(tool_parameters, "job_name") + if action != "list" and not job_name: + raise DiffbotError("job_name is required for this action.") + + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + if action == "list": + jobs = [_job_summary(job) for job in client.crawl_list_jobs()[:100]] + lines = [f"- {job.get('name')}: {(job.get('jobStatus') or {}).get('message', '')}" for job in jobs] + yield from messages( + self, summarize_lines(f"{len(jobs)} crawl job(s).", lines), {"jobs": jobs} + ) + elif action == "status": + job = _job_summary(dict(crawl_get_job(client, job_name))) + message = (job.get("jobStatus") or {}).get("message", "") + yield from messages(self, f"Crawl job {job_name}: {message}".strip(), {"job": job}) + elif action == "results": + limit = int( + bounded_number( + tool_parameters.get("limit") or DEFAULT_RESULT_LIMIT, + label="limit", minimum=1, maximum=RESULT_LIMIT, integer=True, + ) + ) + kind = optional_text(tool_parameters, "result_type").lower() + if kind not in ("", "urls"): + raise DiffbotError("result_type must be empty or urls.") + # Ask for one extra row so "truncated" can be detected. + data = crawl_download(client, job_name, format="json", type=kind or None, num=limit + 1) + rows = data if isinstance(data, list) else [data] + kept = [{k: _cell(v) for k, v in row.items()} if isinstance(row, dict) else row for row in rows[:limit]] + payload = { + "job_name": job_name, + "returned": len(kept), + "truncated": len(rows) > limit, + "results": kept, + } + yield from messages( + self, f"Crawl job {job_name} returned {len(kept)} row(s)" + + (" (truncated)." if payload["truncated"] else "."), payload + ) + else: # delete + client.crawl_delete_job(job_name) + yield from messages(self, f"Deleted crawl job {job_name} and its data.", {"deleted": job_name}) diff --git a/diffbot/tools/crawl_jobs.yaml b/diffbot/tools/crawl_jobs.yaml new file mode 100644 index 0000000..9b56f1c --- /dev/null +++ b/diffbot/tools/crawl_jobs.yaml @@ -0,0 +1,73 @@ +identity: + name: crawl_jobs + author: diffbot + label: + en_US: Crawl jobs +description: + human: + en_US: "List Diffbot crawl jobs, check one's status, fetch its results, or delete it. Delete is destructive: it permanently removes the job and its collected data." + llm: > + Manage Diffbot crawl jobs. action list shows all jobs; status shows one job's progress; + results returns up to 1000 rows (truncated) of its output; delete PERMANENTLY removes the job and its data, + so only use delete when the user explicitly asks for it. job_name is required except for list. +parameters: + - name: action + type: select + required: true + form: llm + default: list + label: + en_US: Action + human_description: + en_US: List, status, results, or delete. DELETE IS DESTRUCTIVE and permanently removes the job and its data. + llm_description: list, status, results, or delete (destructive, irreversible). + options: + - value: list + label: + en_US: List jobs + - value: status + label: + en_US: Status + - value: results + label: + en_US: Results + - value: delete + label: + en_US: Delete (destructive) + - name: job_name + type: string + required: false + form: llm + label: + en_US: Job name + human_description: + en_US: Name of the crawl job. Required for status, results, and delete. + llm_description: Name of the crawl job. Required for status, results, and delete. + - name: result_type + type: select + required: false + form: llm + label: + en_US: Result type + human_description: + en_US: "Results only: extracted objects (default) or the crawled URL report." + llm_description: Leave empty for extracted objects, or urls for the URL report. + options: + - value: urls + label: + en_US: URL report + - name: limit + type: number + required: false + form: form + default: 100 + min: 1 + max: 1000 + label: + en_US: Result limit + human_description: + en_US: "Results only: most rows to return (1-1000). Larger results are truncated." + llm_description: "Results only: most rows to return (1-1000). Larger results are truncated." +extra: + python: + source: tools/crawl_jobs.py diff --git a/diffbot/tools/crawl_start.py b/diffbot/tools/crawl_start.py new file mode 100644 index 0000000..961d07d --- /dev/null +++ b/diffbot/tools/crawl_start.py @@ -0,0 +1,84 @@ +"""Start a Diffbot crawl job. Returns the job name, never results.""" + +from collections.abc import Generator +from typing import Any + +from dify_plugin import Tool +from dify_plugin.entities.tool import ToolInvokeMessage + +from tools.client import DiffbotError, validate_public_http_url +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import crawl_start + +MAX_JOB_NAME = 30 + + +def _int(parameters: dict[str, Any], name: str, *, minimum: int, maximum: int) -> int | None: + value = parameters.get(name) + if value in (None, ""): + return None + return int(bounded_number(value, label=name, minimum=minimum, maximum=maximum, integer=True)) + + +def _seeds(raw: str) -> str: + parts = raw.split() + if not parts: + raise DiffbotError("At least one seed URL is required.") + if len(parts) > 20: + raise DiffbotError("Too many seed URLs (20 maximum).") + checked = [] + for part in parts: + checked.append(validate_public_http_url(part if "://" in part else f"https://{part}")) + return " ".join(checked) + + +class CrawlStartTool(Tool): + def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: + seeds = _seeds(optional_text(tool_parameters, "seeds")) + kwargs: dict[str, Any] = {} + name = optional_text(tool_parameters, "job_name") + if name: + # The crawler rejects names over 30 characters with HTTP 200 + errorCode 505 + # and creates nothing; diffbot 3.2.0 does not surface that, so check here. + if not all(c.isalnum() or c in "-_." for c in name) or len(name) > MAX_JOB_NAME: + raise DiffbotError( + f"job_name may use letters, digits, '-', '_' and '.' ({MAX_JOB_NAME} characters maximum)." + ) + kwargs["job_name"] = name + for param, key, low, high in ( + ("max_to_crawl", "max_to_crawl", 1, 10000), + ("max_to_process", "max_to_process", 1, 10000), + ("hops", "hops", 0, 10), + ): + value = _int(tool_parameters, param, minimum=low, maximum=high) + if value is not None: + kwargs[key] = value + for param in ("restrict_domain", "obey_robots", "use_proxies"): + if isinstance(tool_parameters.get(param), bool): + kwargs[param] = tool_parameters[param] + delay = tool_parameters.get("crawl_delay") + if delay not in (None, ""): + kwargs["crawl_delay"] = float(bounded_number(delay, label="crawl_delay", minimum=-1, maximum=3600)) + for param in ("url_crawl_pattern", "url_process_pattern", "custom_headers"): + value = optional_text(tool_parameters, param) + if value: + kwargs[param] = value + api_url = optional_text(tool_parameters, "api_url") + if api_url: + kwargs["api_url"] = validate_public_http_url(api_url) + + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + # watch=False always: polling would outlive the plugin's 300 s call limit. + job_name = crawl_start(client, seeds, **kwargs) + payload = { + "job_name": job_name, + "seeds": seeds.split(), + "next": "Use Crawl jobs with action status, then results, with this job_name.", + } + yield from messages( + self, + f"Started Diffbot crawl job {job_name}. It runs in the background; check it with the Crawl jobs tool.", + payload, + ) diff --git a/diffbot/tools/crawl_start.yaml b/diffbot/tools/crawl_start.yaml new file mode 100644 index 0000000..8a5728d --- /dev/null +++ b/diffbot/tools/crawl_start.yaml @@ -0,0 +1,148 @@ +identity: + name: crawl_start + author: diffbot + label: + en_US: Start crawl +description: + human: + en_US: Start a Diffbot crawl of one or more sites. This returns a job name immediately; it does not wait for pages. Check progress and fetch results with the Crawl jobs tool. + llm: > + Start a Diffbot crawl job on public http(s) seed URLs. Returns a job_name only, never results, + because crawls run in the background. Afterwards call crawl_jobs with action status until the job + is done, then action results. Do not use for private or local hosts. +parameters: + - name: seeds + type: string + required: true + form: llm + label: + en_US: Seed URLs + human_description: + en_US: One or more public URLs to start from, separated by spaces. Bare domains get https://. + llm_description: Space-separated public http(s) URLs to crawl. + - name: job_name + type: string + required: false + form: llm + label: + en_US: Job name + human_description: + en_US: Optional name (letters, digits, - _ .; 30 characters maximum). Defaults to crawl-. + llm_description: Optional job name, at most 30 characters (letters, digits, - _ .). Leave empty to auto-generate. + - name: max_to_crawl + type: number + required: false + form: form + default: 100 + min: 1 + max: 10000 + label: + en_US: Max pages to crawl + human_description: + en_US: Most pages to fetch (1-10000). + llm_description: Most pages to fetch (1-10000). + - name: max_to_process + type: number + required: false + form: form + default: 100 + min: 1 + max: 10000 + label: + en_US: Max pages to process + human_description: + en_US: Most pages to extract (1-10000). + llm_description: Most pages to extract (1-10000). + - name: hops + type: number + required: false + form: form + default: 2 + min: 0 + max: 10 + label: + en_US: Link hops + human_description: + en_US: How many links deep to follow from the seeds (0-10). + llm_description: How many links deep to follow from the seeds (0-10). + - name: restrict_domain + type: boolean + required: false + form: form + default: true + label: + en_US: Stay on seed domains + human_description: + en_US: Only follow links on the seed domains. + llm_description: Only follow links on the seed domains. + - name: url_crawl_pattern + type: string + required: false + form: form + label: + en_US: URL crawl pattern + human_description: + en_US: Only follow URLs containing one of these strings (Diffbot pattern syntax). + llm_description: Only follow URLs containing one of these strings (Diffbot pattern syntax). + - name: url_process_pattern + type: string + required: false + form: form + label: + en_US: URL process pattern + human_description: + en_US: Only extract URLs containing one of these strings. + llm_description: Only extract URLs containing one of these strings. + - name: crawl_delay + type: number + required: false + form: form + default: -1 + min: -1 + max: 3600 + label: + en_US: Crawl delay (s) + human_description: + en_US: Seconds between requests; -1 uses the Diffbot default. + llm_description: Seconds between requests; -1 uses the Diffbot default. + - name: obey_robots + type: boolean + required: false + form: form + default: false + label: + en_US: Obey robots.txt + human_description: + en_US: Respect robots.txt. + llm_description: Respect robots.txt. + - name: use_proxies + type: boolean + required: false + form: form + default: false + label: + en_US: Use proxies + human_description: + en_US: Route requests through Diffbot proxies. + llm_description: Route requests through Diffbot proxies. + - name: api_url + type: string + required: false + form: form + label: + en_US: Extraction API URL + human_description: + en_US: Optional Diffbot API URL to run on each page (public https only). + llm_description: Optional Diffbot API URL to run on each page (public https only). + - name: custom_headers + type: string + required: false + form: form + label: + en_US: Custom headers + human_description: + en_US: "Optional header to send with each request. Do not put secrets here: the crawler sends it to the sites." + llm_description: "Optional header to send with each request. Do not put secrets here: the crawler sends it to the sites." +extra: + python: + source: tools/crawl_start.py diff --git a/diffbot/tools/enhance_entity.py b/diffbot/tools/enhance_entity.py deleted file mode 100644 index 983c1bd..0000000 --- a/diffbot/tools/enhance_entity.py +++ /dev/null @@ -1,62 +0,0 @@ -"""Enhance a person or organization from partial public identifiers.""" - -from collections.abc import Generator -from typing import Any - -from dify_plugin import Tool -from dify_plugin.entities.tool import ToolInvokeMessage - -from tools.client import DiffbotError, validate_public_http_url -from tools.common import make_client, messages, optional_text, result_size -from tools.shape import summarize_lines, trim_enhance - - -class EnhanceEntityTool(Tool): - def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: - entity_type = optional_text(tool_parameters, "entity_type") - if entity_type not in {"Organization", "Person"}: - raise DiffbotError("entity_type must be Organization or Person.") - name = optional_text(tool_parameters, "name") - url = optional_text(tool_parameters, "url") - location = optional_text(tool_parameters, "location") - employer = optional_text(tool_parameters, "employer") - title = optional_text(tool_parameters, "title") - if entity_type == "Organization" and (employer or title): - raise DiffbotError("employer and title apply only when entity_type is Person.") - if not any((name, url, employer)): - raise DiffbotError("Provide a name, a url, or, for a person, an employer.") - if url: - url = validate_public_http_url(url) - - params: dict[str, Any] = { - "type": entity_type, - "size": result_size(tool_parameters, default=1, maximum=3), - } - if name: - params["name"] = name - if url: - params["url"] = url - if location: - params["location"] = location - if employer: - params["employer"] = employer - if title: - params["title"] = title - - data = make_client(self.runtime.credentials).enhance(params) - payload = trim_enhance(data, entity_type) - lines = [] - for index, match in enumerate(payload["matches"], start=1): - entity = match.get("entity") or {} - score = match.get("score") - score_text = f" (score {score})" if score is not None else "" - lines.append(f"{index}. {entity.get('name') or 'Unnamed match'}{score_text}") - if entity.get("homepageUri"): - lines.append(f" {entity['homepageUri']}") - if entity.get("diffbotUri"): - lines.append(f" {entity['diffbotUri']}") - summary = summarize_lines( - f"Enhance returned {len(payload['matches'])} {entity_type.lower()} match(es).", - lines, - ) - yield from messages(self, summary, payload) diff --git a/diffbot/tools/enhance_entity.yaml b/diffbot/tools/enhance_entity.yaml deleted file mode 100644 index 2178493..0000000 --- a/diffbot/tools/enhance_entity.yaml +++ /dev/null @@ -1,91 +0,0 @@ -identity: - name: enhance_entity - author: diffbot - label: - en_US: Enhance entity -description: - human: - en_US: Match a person or organization from a name, homepage, or employer and return the Knowledge Graph record Diffbot considers the best fit. - llm: > - Call Diffbot Enhance to resolve one person or organization from partial public identifiers. - entity_type is Organization or Person. Provide name, url, or for a person an employer. - employer and title are valid only for Person. Email, phone, and IP are intentionally not accepted. - size is 1 to 3. The response includes the match score and a trimmed entity record. - This does not refresh or recrawl origins. -parameters: - - name: entity_type - type: select - required: true - form: llm - label: - en_US: Entity type - human_description: - en_US: Whether to match an organization or a person. - llm_description: Organization or Person. - options: - - value: Organization - label: - en_US: Organization - - value: Person - label: - en_US: Person - - name: name - type: string - required: false - form: llm - label: - en_US: Name - human_description: - en_US: Name of the organization or person. - llm_description: Entity name. More specific names match better. - - name: url - type: string - required: false - form: llm - label: - en_US: URL - human_description: - en_US: Public homepage or profile URL. Local and private hosts are rejected. - llm_description: Public http(s) homepage or profile URL. - - name: location - type: string - required: false - form: llm - label: - en_US: Location - human_description: - en_US: Optional city, region, or country that disambiguates the name. - llm_description: Free-text location used only to disambiguate the match. - - name: employer - type: string - required: false - form: llm - label: - en_US: Employer - human_description: - en_US: Employer name. Only for a person. - llm_description: Employer name. Valid only when entity_type is Person. - - name: title - type: string - required: false - form: llm - label: - en_US: Title - human_description: - en_US: Job title. Only for a person, and only together with a name, URL, or employer. - llm_description: Job title. Valid only when entity_type is Person. Not sufficient on its own. - - name: size - type: number - required: false - form: llm - default: 1 - min: 1 - max: 3 - label: - en_US: Maximum matches - human_description: - en_US: How many candidate matches to return, from 1 to 3. Defaults to 1. - llm_description: Match cap from 1 to 3. Default 1. Each extra match can spend more credits. -extra: - python: - source: tools/enhance_entity.py diff --git a/diffbot/tools/extract.py b/diffbot/tools/extract.py index 0ed7a53..a11ba70 100644 --- a/diffbot/tools/extract.py +++ b/diffbot/tools/extract.py @@ -1,21 +1,80 @@ """Diffbot page extraction tool.""" +import re from collections.abc import Generator from typing import Any from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.client import DiffbotError, validate_public_http_url -from tools.common import make_client, messages, optional_text +from tools.client import DiffbotError, normalize_page_url +from tools.common import make_client, mapped_errors, messages, optional_text +from tools.dql_builder import bounded_number +from tools.sdk import as_dict, extract from tools.shape import summarize_lines, trim_extract PAGE_TYPES = {"auto", "article", "product", "discussion", "image", "video", "list", "event"} +FIELD_PATTERN = re.compile(r"^[A-Za-z0-9_.*,\-]{1,500}$") +# Value sets from the SDK 3.3.0 types (ExtractMode, ExtractFallback, UseProxy, NaturalLanguageField). +MODES = {"article", "product", "discussion", "image", "video", "list", "event", "llm"} +FALLBACKS = {"article", "product", "discussion", "image", "video", "list", "event"} +PROXIES = {"default", "none"} +NATURAL_LANGUAGE = { + "entities", "sentiment", "summary", "facts", "openFacts", "records", "categories", "sentences", "language", +} +_NUMBERS = ( + # name, minimum, maximum, integer + ("max_pages", 1, 100, True), + ("summary_num_sentences", 1, 20, True), + ("max_tags", 1, 100, True), + ("tag_confidence", 0, 1, False), + ("category_confidence", 0, 1, False), +) +_BOOLEANS = ("discussion", "paging") + + +def _extract_options(parameters: dict[str, Any]) -> dict[str, Any]: + """Caller-set Extract options, named as in diffbot 3.3.0; unset ones are omitted.""" + options: dict[str, Any] = {} + fields = optional_text(parameters, "fields").replace(" ", "") + if fields: + if not FIELD_PATTERN.fullmatch(fields): + raise DiffbotError("fields must be a comma-separated list of field names.") + options["fields"] = [part for part in fields.split(",") if part] + for name, allowed in (("mode", MODES), ("fallback", FALLBACKS), ("use_proxy", PROXIES)): + value = optional_text(parameters, name) + if value: + if value not in allowed: + raise DiffbotError(f"{name} must be one of: {', '.join(sorted(allowed))}.") + options[name] = value + timeout = parameters.get("timeout") + if timeout not in (None, ""): + # Milliseconds, the Extract API's unit. The plugin runtime kills calls at 300 s. + options["timeout"] = int( + bounded_number(timeout, label="timeout", minimum=1000, maximum=250000, integer=True) + ) + for name, low, high, integer in _NUMBERS: + value = parameters.get(name) + if value not in (None, ""): + number = bounded_number(value, label=name, minimum=low, maximum=high, integer=integer) + options[name] = int(number) if integer else float(number) + for name in _BOOLEANS: + if parameters.get(name) is True: + options[name] = True # false is the API default, so only send true + natural = [part for part in optional_text(parameters, "natural_language").replace(" ", "").split(",") if part] + if natural: + bad = [part for part in natural if part not in NATURAL_LANGUAGE] + if bad: + raise DiffbotError(f"natural_language must use: {', '.join(sorted(NATURAL_LANGUAGE))}.") + options["natural_language"] = natural + return options + + class ExtractTool(Tool): def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessage, None, None]: - url = validate_public_http_url(optional_text(tool_parameters, "url")) + url = normalize_page_url(optional_text(tool_parameters, "url")) page_type = optional_text(tool_parameters, "page_type").lower() or "auto" if page_type not in PAGE_TYPES: raise DiffbotError( @@ -25,11 +84,13 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag if output_format not in {"markdown", "json"}: raise DiffbotError("format must be markdown or json.") - data = make_client(self.runtime.credentials).extract( - url, - page_type=page_type, - markdown=(output_format == "markdown"), - ) + options = _extract_options(tool_parameters) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + data = as_dict( + extract(client, url, "analyze" if page_type == "auto" else page_type, output_format, options), + "extract", + ) payload = trim_extract(data) payload["url"] = url payload["page_type"] = page_type diff --git a/diffbot/tools/extract.yaml b/diffbot/tools/extract.yaml index efdda6e..13a4f49 100644 --- a/diffbot/tools/extract.yaml +++ b/diffbot/tools/extract.yaml @@ -20,7 +20,7 @@ parameters: label: en_US: Page URL human_description: - en_US: Public http or https URL. Local and private hosts are rejected. + en_US: Public http or https URL; a bare domain like example.com gets https://. Local and private hosts are rejected. llm_description: Absolute public http or https URL of the page to extract. placeholder: en_US: https://example.com/story @@ -76,6 +76,191 @@ parameters: - value: json label: en_US: JSON + - name: fields + type: string + required: false + form: llm + label: + en_US: Fields + human_description: + en_US: Comma-separated extra fields to return, for example links,meta,tags. Leave empty for the defaults. + llm_description: Optional comma-separated Diffbot extract fields, for example links,meta or sentiment. + - name: timeout + type: number + required: false + form: form + min: 1000 + max: 250000 + label: + en_US: Extraction timeout (ms) + human_description: + en_US: Milliseconds Diffbot may spend on the page. Default 30000. + llm_description: Leave empty. + - name: discussion + type: boolean + required: false + form: form + default: false + label: + en_US: Include discussion + human_description: + en_US: "Article/product: also extract comments and reviews." + llm_description: Leave unset. + - name: paging + type: boolean + required: false + form: form + default: false + label: + en_US: Follow pagination + human_description: + en_US: Stitch multi-page articles together. + llm_description: Leave unset. + - name: max_pages + type: number + required: false + form: form + min: 1 + max: 100 + label: + en_US: Max pages + human_description: + en_US: With pagination on, the most pages to follow (1-100). + llm_description: Leave empty. + - name: natural_language + type: string + required: false + form: form + label: + en_US: Natural language + human_description: + en_US: "Comma-separated natural-language fields to add (article): entities, sentiment, summary, facts, openFacts, records, categories, sentences, language." + llm_description: Leave empty. + - name: summary_num_sentences + type: number + required: false + form: form + min: 1 + max: 20 + label: + en_US: Summary sentences + human_description: + en_US: Sentences in the summary when natural language includes summary (1-20). + llm_description: Leave empty. + - name: max_tags + type: number + required: false + form: form + min: 1 + max: 100 + label: + en_US: Max tags + human_description: + en_US: Most tags to return per page (1-100). + llm_description: Leave empty. + - name: tag_confidence + type: number + required: false + form: form + min: 0 + max: 1 + label: + en_US: Tag confidence + human_description: + en_US: Minimum tag confidence, 0 to 1. + llm_description: Leave empty. + - name: category_confidence + type: number + required: false + form: form + min: 0 + max: 1 + label: + en_US: Category confidence + human_description: + en_US: Minimum category confidence, 0 to 1. + llm_description: Leave empty. + - name: mode + type: select + required: false + form: form + label: + en_US: Mode + human_description: + en_US: Restrict Analyze to one Extract API, or llm for markdown. Overrides the Markdown format's default (llm). + llm_description: Leave empty. + options: + - value: article + label: + en_US: article + - value: product + label: + en_US: product + - value: discussion + label: + en_US: discussion + - value: image + label: + en_US: image + - value: video + label: + en_US: video + - value: list + label: + en_US: list + - value: event + label: + en_US: event + - value: llm + label: + en_US: llm + - name: fallback + type: select + required: false + form: form + label: + en_US: Fallback page type + human_description: + en_US: API to use when Analyze classifies the page as other. + llm_description: Leave empty. + options: + - value: article + label: + en_US: article + - value: product + label: + en_US: product + - value: discussion + label: + en_US: discussion + - value: image + label: + en_US: image + - value: video + label: + en_US: video + - value: list + label: + en_US: list + - value: event + label: + en_US: event + - name: use_proxy + type: select + required: false + form: form + label: + en_US: Proxy + human_description: + en_US: Fetch through Diffbot's datacenter proxy (default) or explicitly none. + llm_description: Leave empty. + options: + - value: default + label: + en_US: default + - value: none + label: + en_US: none extra: python: source: tools/extract.py diff --git a/diffbot/tools/resolve_entities.py b/diffbot/tools/resolve_entities.py index d5716d8..679e506 100644 --- a/diffbot/tools/resolve_entities.py +++ b/diffbot/tools/resolve_entities.py @@ -7,7 +7,7 @@ from dify_plugin.entities.tool import ToolInvokeMessage from tools.client import DiffbotError -from tools.common import make_client, messages, optional_text +from tools.common import as_dict, make_client, mapped_errors, messages, optional_text from tools.dql_builder import validate_language from tools.shape import summarize_lines, trim_resolved_entities @@ -23,7 +23,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag if lang != "auto": lang = validate_language(lang) - data = make_client(self.runtime.credentials).resolve_entities(text, lang=lang) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + data = as_dict(client.entities(text, lang=lang), "entity resolution") payload = trim_resolved_entities(data) lines = [] for entity in payload["entities"]: diff --git a/diffbot/tools/sdk.py b/diffbot/tools/sdk.py new file mode 100644 index 0000000..18f569e --- /dev/null +++ b/diffbot/tools/sdk.py @@ -0,0 +1,308 @@ +"""Builds the Diffbot SDK client and adapts it to this plugin's policy. + +The plugin runs on the ``diffbot`` Python SDK (sync ``Diffbot`` client). Everything +the SDK does not expose yet is reached through the public ``client.request()`` +transport, never through private attributes, and each such helper feature-detects +the real method so it drops out when the SDK ships it: + +* ``verify_token`` -> diffbot 3.3.0 (fallback: ``GET /v4/account?days=1``) +* ``crawl_download`` -> diffbot 3.3.0 (fallback: ``GET /v3/crawl/data``) +* extract options -> diffbot 3.3.0 (fallback: ``GET /v3/{api}`` with the same params) + +TODO: when diffbot 3.2.0 / 3.3.0 are released, pin ``diffbot`` in +``requirements.txt`` / ``pyproject.toml`` and delete the fallbacks. +""" + +from __future__ import annotations + +import csv +import inspect +import io +import json +import urllib.request +from collections.abc import Iterator, Mapping +from contextlib import contextmanager +from typing import Any, cast + +import httpx +from diffbot import ( + APIError, + AuthError, + Diffbot, + DiffbotError as SdkError, + ExtractionError, + JobError, + NotFoundError, + RateLimitError, + ValidationError, +) + +from tools.client import ( + DiffbotError, + check_public_host, + normalize_token, + redact, +) +from tools.sdk_types import CrawlJob, ExtractResponse + +DEFAULT_TIMEOUT = 30.0 +MIN_TIMEOUT = 5.0 +MAX_TIMEOUT = 300.0 # dify_plugin's request ceiling is 300 s + +INVALID_TOKEN = "Invalid or unauthorized Diffbot token." + + +class GuardedTransport(httpx.BaseTransport): + """Wraps another transport and refuses private/loopback/metadata hosts and + non-http(s) schemes before any socket is opened. + + With no ``inner`` transport it honors ``HTTP(S)_PROXY`` / ``ALL_PROXY`` / + ``NO_PROXY`` like the old ``requests`` client did: httpx ignores the environment + once an explicit transport is passed, which would break proxied Dify deployments. + """ + + def __init__(self, inner: httpx.BaseTransport | None = None) -> None: + self._proxied: dict[str, httpx.BaseTransport] = {} + if inner is None: + proxies = urllib.request.getproxies() + for scheme in ("http", "https"): + proxy = proxies.get(scheme) or proxies.get("all") + if proxy: + self._proxied[scheme] = httpx.HTTPTransport(proxy=proxy) + self._inner = inner or httpx.HTTPTransport() + + def _transport_for(self, url: httpx.URL) -> httpx.BaseTransport: + proxied = self._proxied.get(url.scheme) + if proxied is None or urllib.request.proxy_bypass(url.host): + return self._inner + return proxied + + def handle_request(self, request: httpx.Request) -> httpx.Response: + if request.url.scheme not in ("http", "https"): + raise DiffbotError("Request scheme is not allowed.") + check_public_host(request.url.host or "") + return self._transport_for(request.url).handle_request(request) + + def close(self) -> None: + self._inner.close() + for transport in self._proxied.values(): + transport.close() + + +def parse_timeout(value: Any) -> float: + if value is None or (isinstance(value, str) and not value.strip()): + return DEFAULT_TIMEOUT + try: + seconds = float(value) + except (TypeError, ValueError): + raise DiffbotError("Request timeout must be a number of seconds.") from None + if not MIN_TIMEOUT <= seconds <= MAX_TIMEOUT: + raise DiffbotError(f"Request timeout must be between {MIN_TIMEOUT:g} and {MAX_TIMEOUT:g} seconds.") + return seconds + + +def make_client(credentials: Mapping[str, Any] | None, *, transport: httpx.BaseTransport | None = None) -> Diffbot: + """Validate the credential form and return an SDK client behind the host guard.""" + credentials = credentials or {} + raw = credentials.get("diffbot_token", "") + token = normalize_token(raw if isinstance(raw, str) else "") + timeout = parse_timeout(credentials.get("request_timeout")) + return Diffbot(token, timeout=timeout, transport=GuardedTransport(transport)) + + +def api_error_message(exc: BaseException, token: str) -> str: + """Safe, user-facing text for an SDK/HTTP failure. Never includes the token.""" + if isinstance(exc, DiffbotError): + return redact(str(exc), token) + if isinstance(exc, AuthError): + return INVALID_TOKEN + if isinstance(exc, RateLimitError): + wait = f" Retry after {redact(str(exc.retry_after), token)} seconds." if exc.retry_after else " Retry later." + return f"Diffbot rate limit exceeded.{wait}" + if isinstance(exc, ExtractionError): + if exc.error_code in (401, 403): + return INVALID_TOKEN + return f"Diffbot could not extract the page ({exc.error_code}): {redact(exc.error, token)[:300]}" + if isinstance(exc, JobError): + # JobError carries plain text, so APIError.message is empty; show the body. + return f"Diffbot crawl job error: {redact(exc.body or 'job failed', token)[:300]}" + if isinstance(exc, NotFoundError): + return "Diffbot did not find that resource." + if isinstance(exc, APIError): + detail = redact(exc.message or "request failed", token)[:300] + return f"Diffbot API error {exc.status_code}: {detail}" + if isinstance(exc, httpx.TimeoutException): + return "The Diffbot request timed out." + if isinstance(exc, (httpx.HTTPError, SdkError)): + return "The Diffbot request failed before a response was received." + return "The Diffbot request failed." + + +@contextmanager +def mapped_errors(client: Diffbot) -> Iterator[None]: + """Translate SDK/httpx errors to plugin-safe ``DiffbotError`` messages.""" + try: + yield + except DiffbotError: + raise + except ValidationError as exc: + raise DiffbotError(redact(str(exc), client.token)) from None + except json.JSONDecodeError: + raise DiffbotError("Diffbot returned a non-JSON response.") from None + except (SdkError, httpx.HTTPError) as exc: + raise DiffbotError(api_error_message(exc, client.token)) from None + + +# ---------------------------------------------------------------- credential check + + +def verify_token(client: Diffbot) -> bool: + """True if the token is accepted. Never returns the account payload (it holds the token). + + Only an auth failure means False; other errors propagate. + """ + native = getattr(client, "verify_token", None) + if callable(native): + return bool(native()) + try: + client.request("GET", f"{client.api_base}/v4/account", params={"token": client.token, "days": 1}) + except AuthError: + return False + return True + + +# ---------------------------------------------------------------- extract + +EXTRACT_OPTION_WIRE = { + "timeout": "timeout", + "mode": "mode", + "fields": "fields", + "fallback": "fallback", + "discussion": "discussion", + "paging": "paging", + "max_pages": "maxPages", + "natural_language": "naturalLanguage", + "summary_num_sentences": "summaryNumSentences", + "max_tags": "maxTags", + "tag_confidence": "tagConfidence", + "category_confidence": "categoryConfidence", + "use_proxy": "useProxy", +} +_LIST_OPTIONS = {"fields", "natural_language"} + + +def _supports_extract_options(client: Diffbot) -> bool: + try: + params = inspect.signature(client.extract).parameters + except (TypeError, ValueError): + return False + return "mode" in params or any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()) + + +def _wire_value(value: Any) -> Any: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (list, tuple)): + return ",".join(str(item) for item in value) + return value + + +def extract(client: Diffbot, url: str, api: str, fmt: str, options: Mapping[str, Any]) -> ExtractResponse: + """Extract ``url``. ``options`` use the SDK 3.3.0 keyword names (unset keys omitted).""" + options = {key: value for key, value in options.items() if value is not None} + if not options or _supports_extract_options(client): + return cast(ExtractResponse, client.extract(url, api, fmt, **options)) + # Fallback for SDKs without extract options: same request the SDK builds. + params: dict[str, Any] = {"token": client.token, "url": url, "timeout": 30000} + if fmt == "markdown": + params["mode"] = "llm" + for key, value in options.items(): + params[EXTRACT_OPTION_WIRE[key]] = _wire_value(value) + timeout = params.get("timeout") + per_call = (float(timeout) / 1000.0 + 10.0) if isinstance(timeout, (int, float)) else None + response = client.request("GET", f"{client.analyze_url}/{api}", params=params, timeout=per_call) + data = response.json() + if isinstance(data, dict) and "errorCode" in data: + raise ExtractionError(data["errorCode"], data.get("error", "")) + return cast(ExtractResponse, data) + + +# ---------------------------------------------------------------- crawl + + +def crawl_start(client: Diffbot, seeds: str, **kwargs: Any) -> str: + """Create a crawl job (never watches) and return its name.""" + for event in client.crawl(seeds, watch=False, **kwargs): + name = (event.details or {}).get("job_name") + if name: + return str(name) + raise DiffbotError("Diffbot did not report a crawl job name.") + + +def crawl_get_job(client: Diffbot, job_name: str) -> CrawlJob: + job = client.crawl_get_job(job_name) + if not job: + raise DiffbotError("Diffbot did not find that crawl job.") + return cast(CrawlJob, job) + + +def _download_text(result: Any) -> Any: + """Coerce a ``crawl_download`` return value (Response, bytes, str, parsed) to text/objects.""" + if isinstance(result, (list, dict)): + return result + if hasattr(result, "text"): + return result.text + if isinstance(result, (bytes, bytearray)): + return bytes(result).decode("utf-8", errors="replace") + return str(result) + + +def crawl_download( + client: Diffbot, job_name: str, *, format: str = "json", type: str | None = None, num: int | None = None +) -> Any: + """Fetch crawl output as parsed rows/objects (a list, or a dict for JSON objects).""" + if type == "urls": + format = "csv" # the URL report is CSV; the JSON decoder would reject it + native = getattr(client, "crawl_download", None) + if callable(native): + raw = native(job_name, format=format, type=type, num=num) + else: + params: dict[str, Any] = {"token": client.token, "name": job_name} + if format and format != "json": + params["format"] = format + if type: + params["type"] = type + if num is not None: + params["num"] = num + raw = client.request("GET", f"{client.crawler_url}/data", params=params, follow_redirects=True) + data = _download_text(raw) + if not isinstance(data, str): + return data + if not data.strip(): + return [] + try: + parsed = json.loads(data) + except ValueError: + return list(csv.DictReader(io.StringIO(data))) + if isinstance(parsed, dict) and parsed.get("errorCode"): + # Same as diffbot 3.3.0: e.g. an unknown job is HTTP 200 + errorCode. + raise JobError(job_name, f"Crawl data download failed: {parsed.get('error') or 'unexpected response'}") + return parsed + + +def as_dict(value: Any, what: str) -> dict[str, Any]: + if not isinstance(value, dict): + raise DiffbotError(f"Diffbot {what} returned an unexpected response.") + return value + + +__all__ = [ + "GuardedTransport", + "make_client", + "mapped_errors", + "verify_token", + "extract", + "crawl_start", + "crawl_get_job", + "crawl_download", +] diff --git a/diffbot/tools/sdk_types.py b/diffbot/tools/sdk_types.py new file mode 100644 index 0000000..2287883 --- /dev/null +++ b/diffbot/tools/sdk_types.py @@ -0,0 +1,37 @@ +"""Response shapes the plugin reads from the Diffbot SDK. + +TODO(diffbot 3.3.0): the SDK ships response TypedDicts in ``diffbot.types``. Until +the pin requires it, these local ``total=False`` equivalents describe only the +fields this plugin touches. Delete the fallback once ``diffbot>=3.3.0`` is pinned. +""" + +from __future__ import annotations + +from typing import Any, TypedDict + +try: # diffbot >= 3.3.0 ships these; prefer them. + from diffbot.types import CrawlJob, ExtractResponse # type: ignore[attr-defined] + + SDK_TYPES = True +except ImportError: # diffbot 3.2.x: local equivalents below + SDK_TYPES = False + + +if not SDK_TYPES: + + class ExtractResponse(TypedDict, total=False): # type: ignore[no-redef] + objects: list[dict[str, Any]] + humanLanguage: str + errorCode: int + error: str + + class CrawlJobStatus(TypedDict, total=False): + status: int + message: str + + class CrawlJob(TypedDict, total=False): # type: ignore[no-redef] + name: str + jobStatus: CrawlJobStatus + pageCrawlSuccesses: int + pageProcessSuccesses: int + objectsFound: int diff --git a/diffbot/tools/search_news.py b/diffbot/tools/search_news.py index 7cdf6c7..ccbdae9 100644 --- a/diffbot/tools/search_news.py +++ b/diffbot/tools/search_news.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_news_query from tools.shape import summarize_lines, trim_kg @@ -23,11 +23,10 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "until": optional_text(tool_parameters, "until"), } query = build_news_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) + size = result_size(tool_parameters, default=5, maximum=100) include_text = bool(tool_parameters.get("include_text")) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "article", include_text=include_text) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "article", include_text=include_text, limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): headline = row.get("title") or "Untitled article" diff --git a/diffbot/tools/search_news.yaml b/diffbot/tools/search_news.yaml index ea1249c..a204347 100644 --- a/diffbot/tools/search_news.yaml +++ b/diffbot/tools/search_news.yaml @@ -92,12 +92,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many articles to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many articles to return, from 1 to 100. + llm_description: Result cap from 1 to 100. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_news.py diff --git a/diffbot/tools/search_organizations.py b/diffbot/tools/search_organizations.py index db1a113..55a41b0 100644 --- a/diffbot/tools/search_organizations.py +++ b/diffbot/tools/search_organizations.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_organizations_query from tools.shape import summarize_lines, trim_kg @@ -26,10 +26,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "is_public": optional_text(tool_parameters, "is_public").lower() or "any", } query = build_organizations_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "organization") + size = result_size(tool_parameters, default=5, maximum=100) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "organization", limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): location = row.get("location") or {} diff --git a/diffbot/tools/search_organizations.yaml b/diffbot/tools/search_organizations.yaml index d6ef184..291140b 100644 --- a/diffbot/tools/search_organizations.yaml +++ b/diffbot/tools/search_organizations.yaml @@ -123,12 +123,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many organizations to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many organizations to return, from 1 to 100. + llm_description: Result cap from 1 to 100. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_organizations.py diff --git a/diffbot/tools/search_people.py b/diffbot/tools/search_people.py index 8bf281f..44272df 100644 --- a/diffbot/tools/search_people.py +++ b/diffbot/tools/search_people.py @@ -6,7 +6,7 @@ from dify_plugin import Tool from dify_plugin.entities.tool import ToolInvokeMessage -from tools.common import make_client, messages, optional_text, result_size +from tools.common import messages, optional_text, result_offset, result_size, run_dql from tools.dql_builder import build_people_query from tools.shape import summarize_lines, trim_kg @@ -21,10 +21,9 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag "country": optional_text(tool_parameters, "country"), } query = build_people_query(parameters) - size = result_size(tool_parameters, default=5, maximum=10) - data = make_client(self.runtime.credentials).dql(query, size=size) - data["dql"] = query - payload = trim_kg(data, "person") + size = result_size(tool_parameters, default=5, maximum=100) + data = run_dql(self.runtime.credentials, query, size=size, offset=result_offset(tool_parameters)) + payload = trim_kg(data, "person", limit=size) lines = [] for index, row in enumerate(payload["results"], start=1): jobs = row.get("employments") or [] diff --git a/diffbot/tools/search_people.yaml b/diffbot/tools/search_people.yaml index 2f3c388..195e4af 100644 --- a/diffbot/tools/search_people.yaml +++ b/diffbot/tools/search_people.yaml @@ -64,12 +64,24 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 100 label: en_US: Maximum results human_description: - en_US: How many people to return, from 1 to 10. - llm_description: Result cap from 1 to 10. Default 5. + en_US: How many people to return, from 1 to 100. + llm_description: Result cap from 1 to 100. Default 5. + - name: offset + type: number + required: false + form: form + default: 0 + min: 0 + max: 100000 + label: + en_US: Offset + human_description: + en_US: Number of results to skip, for paging through large result sets. + llm_description: Number of results to skip for paging. Default 0. extra: python: source: tools/search_people.py diff --git a/diffbot/tools/search_web.py b/diffbot/tools/search_web.py index aff70ef..1616bb7 100644 --- a/diffbot/tools/search_web.py +++ b/diffbot/tools/search_web.py @@ -7,7 +7,7 @@ from dify_plugin.entities.tool import ToolInvokeMessage from tools.client import DiffbotError -from tools.common import make_client, messages, optional_text +from tools.common import as_dict, make_client, mapped_errors, messages, optional_text from tools.dql_builder import bounded_number from tools.shape import summarize_lines, trim_web_search @@ -32,32 +32,34 @@ def _invoke(self, tool_parameters: dict[str, Any]) -> Generator[ToolInvokeMessag num_results = 5 else: num_results = int( - bounded_number(num_value, label="num_results", minimum=1, maximum=10, integer=True) + bounded_number(num_value, label="num_results", minimum=1, maximum=50, integer=True) ) snippet = optional_text(tool_parameters, "snippet_length").lower() - rerank = None - if not snippet: - snippet = "full" if _is_page_lookup(query) else "short" - if snippet not in SNIPPETS: + if not snippet and _is_page_lookup(query): + snippet = "full" # a lone url: query asks for the stored page + if snippet and snippet not in SNIPPETS: raise DiffbotError("snippet_length must be short, medium, long, or full.") - if snippet == "full": - rerank = "none" max_tokens = None raw_tokens = tool_parameters.get("max_tokens") if raw_tokens not in (None, ""): max_tokens = int( - bounded_number(raw_tokens, label="max_tokens", minimum=500, maximum=8000, integer=True) + bounded_number(raw_tokens, label="max_tokens", minimum=500, maximum=32000, integer=True) ) - data = make_client(self.runtime.credentials).web_search( - query, - num_results=num_results, - snippet_length=snippet, - max_tokens=max_tokens, - rerank=rerank, - ) + client = make_client(self.runtime.credentials) + with client, mapped_errors(client): + # Blank snippet_length is omitted so the API default (long) applies. + data = as_dict( + client.web_search( + query, + num_results=num_results, + max_tokens=max_tokens, + snippet_length=snippet or None, # type: ignore[arg-type] + ), + "web search", + ) payload = trim_web_search(data) lines = [] for index, item in enumerate(payload["results"], start=1): diff --git a/diffbot/tools/search_web.yaml b/diffbot/tools/search_web.yaml index 9bb6a6b..90a5b63 100644 --- a/diffbot/tools/search_web.yaml +++ b/diffbot/tools/search_web.yaml @@ -32,12 +32,12 @@ parameters: form: llm default: 5 min: 1 - max: 10 + max: 50 label: en_US: Number of results human_description: - en_US: How many results to return, from 1 to 10. Defaults to 5. - llm_description: Maximum number of results, from 1 to 10. Default 5. + en_US: How many results to return, from 1 to 50. Defaults to 5. + llm_description: Maximum number of results, from 1 to 50. Default 5. - name: snippet_length type: select required: false @@ -45,10 +45,10 @@ parameters: label: en_US: Snippet length human_description: - en_US: "How much of each page to return. Leave empty for a short passage, or full for the whole indexed page on a url: lookup." + en_US: "How much of each page to return. Leave empty for the Diffbot default (long). A lone url: query returns the full indexed page unless you choose a size." llm_description: > Passage size. short is about 1 KB, then medium, long, and full. - Leave empty to use short, or full automatically when the query is only a url: lookup. + Leave empty for the API default (long), or full automatically when the query is only a url: lookup. options: - value: short label: @@ -67,12 +67,12 @@ parameters: required: false form: llm min: 500 - max: 8000 + max: 32000 label: en_US: Max tokens human_description: en_US: Optional cap on the response size. Diffbot shortens passages to fit. Ignored for a full-page lookup. - llm_description: Optional response token cap from 500 to 8000. Omit it unless the context window is tight. + llm_description: Optional response token cap from 500 to 32000. Omit it unless the context window is tight. extra: python: source: tools/search_web.py diff --git a/diffbot/tools/shape.py b/diffbot/tools/shape.py index 4d43168..3bfe45f 100644 --- a/diffbot/tools/shape.py +++ b/diffbot/tools/shape.py @@ -115,9 +115,9 @@ def trim_article(entity: dict[str, Any], *, include_text: bool) -> dict[str, Any return {key: value for key, value in record.items() if value not in (None, "", [], {})} -def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False) -> dict[str, Any]: +def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False, limit: int = 20) -> dict[str, Any]: rows = [] - for item in (data.get("data") or [])[:20]: + for item in (data.get("data") or [])[:limit]: entity = item.get("entity") if isinstance(item, dict) and "entity" in item else item if not isinstance(entity, dict): continue @@ -135,17 +135,6 @@ def trim_kg(data: dict[str, Any], kind: str, *, include_text: bool = False) -> d } -def trim_enhance(data: dict[str, Any], kind: str) -> dict[str, Any]: - matches = [] - for item in (data.get("data") or [])[:5]: - if not isinstance(item, dict): - continue - entity = item.get("entity") if isinstance(item.get("entity"), dict) else {} - trimmed = trim_organization(entity) if kind == "Organization" else trim_person(entity) - matches.append({"score": item.get("score"), "entity": trimmed}) - return {"hits": data.get("hits"), "matches": matches} - - def trim_extract(data: dict[str, Any]) -> dict[str, Any]: objects = data.get("objects") trimmed = [] diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..7c4b8e6 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,3 @@ +# Dev/test requirements. Not a second pin: the SDK comes from diffbot/requirements.txt. +-r diffbot/requirements.txt +pytest diff --git a/tests/test_sdk_wiring.py b/tests/test_sdk_wiring.py new file mode 100644 index 0000000..58abd7b --- /dev/null +++ b/tests/test_sdk_wiring.py @@ -0,0 +1,350 @@ +"""SDK wiring: credential check, host guard, extract options, crawl tools, feature detection. + +All HTTP is served by httpx.MockTransport; no live Diffbot calls. +""" + +import json + +import httpx +import pytest +import yaml +from diffbot import Diffbot + +from provider.diffbot import DiffbotProvider +from tools import sdk +from tools.client import DiffbotError +from tools.crawl_jobs import CrawlJobsTool +from tools.crawl_start import CrawlStartTool +from tools.extract import ExtractTool +from tools.search_organizations import SearchOrganizationsTool +from tools.search_web import SearchWebTool + +TOKEN = "a" * 32 +CREDS = {"diffbot_token": TOKEN} + + +@pytest.fixture(autouse=True, params=["native", "legacy"]) +def sdk_flavor(request, monkeypatch): + """Run every test against the SDK as installed (3.3.0 methods) and against an SDK + without them, which exercises the plugin's client.request() fallbacks.""" + if request.param == "legacy": + monkeypatch.delattr(Diffbot, "verify_token") + monkeypatch.delattr(Diffbot, "crawl_download") + real_extract = Diffbot.extract + + def old_extract(self, url, api="analyze", fmt="markdown"): + return real_extract(self, url, api, fmt) + + monkeypatch.setattr(Diffbot, "extract", old_extract) + return request.param + + +@pytest.fixture +def serve(monkeypatch): + """Install ``handler(request) -> httpx.Response``; returns the list of seen requests.""" + seen: list[httpx.Request] = [] + + def install(handler): + def wrapped(request): + seen.append(request) + return handler(request) + + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda **_: httpx.MockTransport(wrapped)) + return seen + + return install + + +def _json(messages): + return [m for m in messages if m.type.value == "json"][-1].message.json_object + + +# ------------------------------------------------------------ credentials + + +def test_validation_calls_account_with_days_1_and_never_surfaces_payload(serve): + seen = serve(lambda r: httpx.Response(200, json={"token": TOKEN, "plan": "FREE"})) + DiffbotProvider()._validate_credentials(CREDS) + assert seen[0].url.path == "/v4/account" + assert seen[0].url.params["days"] == "1" + + +def test_validation_rejects_auth_failure_without_echoing_token(serve): + serve(lambda r: httpx.Response(401, json={"error": f"bad {TOKEN}"})) + with pytest.raises(Exception, match="Invalid or unauthorized") as caught: + DiffbotProvider()._validate_credentials(CREDS) + assert TOKEN not in str(caught.value) + + +def test_validation_does_not_treat_server_errors_as_a_bad_token(serve): + serve(lambda r: httpx.Response(400, json={"message": "bad request"})) + with pytest.raises(Exception) as caught: + DiffbotProvider()._validate_credentials(CREDS) + assert "Invalid or unauthorized" not in str(caught.value) + + +def test_non_json_body_is_a_safe_error(serve): + serve(lambda r: httpx.Response(200, text="oops")) + with pytest.raises(DiffbotError, match="non-JSON"): + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x"})) + + +@pytest.mark.parametrize("value", ["abc", "4", "301", "-1"]) +def test_request_timeout_is_validated(value): + with pytest.raises(DiffbotError, match="timeout"): + sdk.make_client({**CREDS, "request_timeout": value}) + + +def test_request_timeout_is_applied(): + client = sdk.make_client({**CREDS, "request_timeout": "45"}) + assert client._http.timeout.read == 45.0 # test-only peek at the httpx client + + +def test_native_verify_token_is_preferred(): + class Fake: + def verify_token(self): + return False + + assert sdk.verify_token(Fake()) is False # type: ignore[arg-type] + + +# ------------------------------------------------------------ host guard + + +def test_transport_blocks_private_hosts_before_any_socket(): + inner_calls = [] + inner = httpx.MockTransport(lambda r: inner_calls.append(r) or httpx.Response(200, json={})) + client = Diffbot(TOKEN, analyze_url="http://169.254.169.254/v3", transport=sdk.GuardedTransport(inner)) + with pytest.raises(DiffbotError, match="not allowed"): + client.extract("https://example.org") + assert inner_calls == [] + + +def test_default_transport_honors_proxy_environment(monkeypatch): + for name in ("HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY", "http_proxy", "https_proxy", "all_proxy", "no_proxy"): + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("HTTPS_PROXY", "http://proxy.example.org:3128") + monkeypatch.setenv("NO_PROXY", "nl.diffbot.com") + guard = sdk.GuardedTransport() + assert guard._transport_for(httpx.URL("https://api.diffbot.com/v3/analyze")) is guard._proxied["https"] + assert guard._transport_for(httpx.URL("https://nl.diffbot.com/v1/")) is guard._inner + assert guard._transport_for(httpx.URL("http://api.diffbot.com/")) is guard._inner # no HTTP_PROXY + guard.close() + + +def test_explicit_inner_transport_ignores_proxy_environment(monkeypatch): + monkeypatch.setenv("HTTPS_PROXY", "http://proxy.example.org:3128") + inner = httpx.MockTransport(lambda r: httpx.Response(200)) + assert sdk.GuardedTransport(inner)._transport_for(httpx.URL("https://api.diffbot.com/")) is inner + + +def test_guard_cannot_be_disabled(): + calls = [] + inner = httpx.MockTransport(lambda r: calls.append(r) or httpx.Response(200, json={"objects": []})) + client = Diffbot(TOKEN, analyze_url="http://10.0.0.5/v3", transport=sdk.GuardedTransport(inner)) + with pytest.raises(Exception): + client.extract("https://example.org") + assert calls == [] + assert not hasattr(sdk, "_as_bool") + + +def test_rate_limit_maps_to_a_safe_message(serve): + serve(lambda r: httpx.Response(429, json={"message": TOKEN}, headers={"retry-after": "0"})) + with pytest.raises(DiffbotError, match="rate limit") as caught: + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x"})) + assert TOKEN not in str(caught.value) + + +# ------------------------------------------------------------ extract + + +def test_extract_normalizes_bare_domains(serve): + seen = serve(lambda r: httpx.Response(200, json={"objects": []})) + list(ExtractTool.from_credentials(CREDS).invoke({"url": "example.org/story"})) + assert seen[0].url.params["url"] == "https://example.org/story" + + +@pytest.mark.parametrize("url", ["localhost", "127.0.0.1/admin", "10.1.2.3", "metadata.google.internal"]) +def test_extract_rejects_private_bare_domains(url): + with pytest.raises(DiffbotError, match="not allowed"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": url})) + + +def test_extract_options_reach_the_request(serve): + seen = serve(lambda r: httpx.Response(200, json={"objects": []})) + params = { + "url": "https://example.org", + "fields": "links, meta", + "timeout": 60000, + "mode": "llm", + "paging": True, + "fallback": "article", + "use_proxy": "none", + "natural_language": "entities, summary", + "max_pages": 3, + "tag_confidence": 0.5, + } + list(ExtractTool.from_credentials(CREDS).invoke(params)) + q = seen[0].url.params + assert q["fields"] == "links,meta" + assert q["timeout"] == "60000" + assert q["mode"] == "llm" + assert q["paging"] == "true" + assert q["maxPages"] == "3" + assert q["tagConfidence"] == "0.5" + assert q["fallback"] == "article" and q["useProxy"] == "none" + assert q["naturalLanguage"] == "entities,summary" + + +def test_extract_option_validation(): + with pytest.raises(DiffbotError, match="fields"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "fields": "a&token=b"})) + with pytest.raises(DiffbotError, match="timeout"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "timeout": 5})) + + +def test_extract_rejects_unknown_option_values(): + with pytest.raises(DiffbotError, match="mode"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "mode": "bogus"})) + with pytest.raises(DiffbotError, match="natural_language"): + list(ExtractTool.from_credentials(CREDS).invoke({"url": "https://example.org", "natural_language": "x"})) + + +def test_extract_passes_options_to_sdks_that_support_them(): + calls = [] + + class Fake: + token = TOKEN + + def extract(self, url, api="analyze", fmt="markdown", *, mode=None, fields=None, timeout=None): + calls.append((url, api, fmt, mode, fields, timeout)) + return {"objects": []} + + sdk.extract(Fake(), "https://x.org", "article", "json", {"mode": "llm", "fields": ["links"], "timeout": 5000}) # type: ignore[arg-type] + assert calls == [("https://x.org", "article", "json", "llm", ["links"], 5000)] + + +# ------------------------------------------------------------ KG / search limits + + +def test_kg_offset_and_large_size_are_sent(serve): + seen = serve(lambda r: httpx.Response(200, json={"hits": 0, "data": []})) + list(SearchOrganizationsTool.from_credentials(CREDS).invoke({"name": "Diffbot", "size": 100, "offset": 200})) + body = json.loads(seen[0].content) + assert body["size"] == 100 and body["from"] == 200 + + +def test_web_search_allows_50_results_and_rejects_more(serve): + seen = serve(lambda r: httpx.Response(200, json={"search_results": []})) + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x", "num_results": 50, "max_tokens": 32000})) + assert seen[0].url.params["size"] == "50" + with pytest.raises(DiffbotError): + list(SearchWebTool.from_credentials(CREDS).invoke({"query": "x", "num_results": 51})) + + +# ------------------------------------------------------------ crawl + + +def test_crawl_start_never_watches_and_returns_the_job_name(serve): + seen = serve(lambda r: httpx.Response(200, json={})) + payload = _json( + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j1", "hops": 1})) + ) + assert payload["job_name"] == "j1" + assert len(seen) == 1 # create only: no status poll, no data fetch + assert seen[0].url.params["seeds"] == "https://example.org" + assert seen[0].url.params["maxHops"] == "1" + + +@pytest.mark.parametrize("seeds", ["http://127.0.0.1", "https://ok.example.org localhost"]) +def test_crawl_start_guards_seeds(seeds): + with pytest.raises(DiffbotError, match="not allowed"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": seeds})) + + +def test_crawl_start_rejects_job_names_over_30_characters(serve): + seen = serve(lambda r: httpx.Response(200, json={})) + with pytest.raises(DiffbotError, match="30 characters"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j" * 31})) + assert seen == [] + payload = _json(list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "job_name": "j" * 30}))) + assert payload["job_name"] == "j" * 30 + + +def test_crawl_results_not_truncated_when_rows_fit(serve): + serve(lambda r: httpx.Response(200, json=[{"title": "a"}, {"title": "b"}])) + payload = _json(_jobs({"action": "results", "job_name": "j1", "limit": 2})) + assert payload["returned"] == 2 and payload["truncated"] is False + + +def test_crawl_start_guards_api_url(): + with pytest.raises(DiffbotError, match="not allowed"): + list(CrawlStartTool.from_credentials(CREDS).invoke({"seeds": "example.org", "api_url": "http://localhost/x"})) + + +def _jobs(parameters): + return list(CrawlJobsTool.from_credentials(CREDS).invoke(parameters)) + + +def test_crawl_jobs_list_and_status(serve): + job = {"name": "j1", "jobStatus": {"status": 9, "message": "done"}, "objectsFound": 4, "secretish": "x"} + serve(lambda r: httpx.Response(200, json={"jobs": [job]})) + assert _json(_jobs({"action": "list"}))["jobs"][0]["name"] == "j1" + status = _json(_jobs({"action": "status", "job_name": "j1"}))["job"] + assert status["objectsFound"] == 4 and "secretish" not in status + + +def test_crawl_jobs_status_for_missing_job(serve): + serve(lambda r: httpx.Response(200, json={"jobs": []})) + with pytest.raises(DiffbotError, match="not find"): + _jobs({"action": "status", "job_name": "nope"}) + + +def test_crawl_results_fall_back_to_the_data_endpoint_and_parse_csv(serve): + csv_text = 'Url,Crawl Status\nhttps://a.example.org,Success\nhttps://b.example.org,Success\n' + seen = serve(lambda r: httpx.Response(200, text=csv_text)) + payload = _json(_jobs({"action": "results", "job_name": "j1", "result_type": "urls", "limit": 1})) + assert seen[0].url.path == "/v3/crawl/data" + assert seen[0].url.params["type"] == "urls" + assert seen[0].url.params["num"] == "2" # one extra row detects truncation + assert payload["returned"] == 1 and payload["truncated"] is True + assert payload["results"][0]["Url"] == "https://a.example.org" + + +def test_crawl_results_use_native_crawl_download_when_available(): + class Fake: + token = TOKEN + + def crawl_download(self, name, *, format="json", type=None, num=None): + return httpx.Response(200, json=[{"title": "t"}]) + + assert sdk.crawl_download(Fake(), "j", num=5) == [{"title": "t"}] # type: ignore[arg-type] + + +def test_crawl_download_job_error_shows_the_reason(serve): + serve(lambda r: httpx.Response(200, json={"errorCode": 500, "error": "No collection record"})) + with pytest.raises(DiffbotError, match="No collection record"): + _jobs({"action": "results", "job_name": "j1"}) + + +def test_crawl_delete_is_not_retried(serve): + seen = serve(lambda r: httpx.Response(503, json={"message": "down"})) + with pytest.raises(DiffbotError): + _jobs({"action": "delete", "job_name": "j1"}) + assert len(seen) == 1 + assert seen[0].url.params["delete"] == "1" + + +def test_crawl_delete_is_labelled_destructive(): + spec = yaml.safe_load(open("diffbot/tools/crawl_jobs.yaml")) + assert "destructive" in spec["description"]["human"]["en_US"].lower() + action = next(p for p in spec["parameters"] if p["name"] == "action") + assert "destructive" in action["human_description"]["en_US"].lower() + + +def test_all_tool_yamls_are_registered_and_exist(): + provider = yaml.safe_load(open("diffbot/provider/diffbot.yaml")) + for path in provider["tools"]: + spec = yaml.safe_load(open(f"diffbot/{path}")) + assert spec["extra"]["python"]["source"].startswith("tools/") + assert {"tools/crawl_start.yaml", "tools/crawl_jobs.yaml"} <= set(provider["tools"]) diff --git a/tests/test_tools_http.py b/tests/test_tools_http.py index cc946d4..8b3ec08 100644 --- a/tests/test_tools_http.py +++ b/tests/test_tools_http.py @@ -2,12 +2,11 @@ import json +import httpx import pytest -import requests from provider.diffbot import DiffbotProvider from tools.client import TOKEN_PATTERN, DiffbotError -from tools.enhance_entity import EnhanceEntityTool from tools.extract import ExtractTool from tools.resolve_entities import ResolveEntitiesTool from tools.search_news import SearchNewsTool @@ -20,38 +19,38 @@ class FakeResponse: - def __init__(self, status_code, payload): - self.status_code = status_code - self._payload = payload - self.text = payload if isinstance(payload, str) else json.dumps(payload) - - def json(self): - if isinstance(self._payload, str): - raise ValueError("not json") - return self._payload - - -class FakeSession: - def __init__(self, responder): - self.responder = responder - self.calls = [] + """(status, payload) pair a responder returns; converted to an httpx.Response.""" - def request(self, method, url, **kwargs): - self.calls.append({"method": method, "url": url, **kwargs}) - return self.responder(method, url, kwargs) + def __init__(self, status_code, payload, headers=None): + self.status_code = status_code + self.payload = payload + self.headers = headers or {} @pytest.fixture def session_factory(monkeypatch): - holder = {} + """Serve every SDK request from ``responder(method, url, kwargs)`` via httpx.MockTransport. - def install(responder): - def factory(): - session = FakeSession(responder) - holder["session"] = session - return session + ``kwargs`` has ``params`` (query dict), ``headers`` and ``json`` (parsed body), so the + real SDK client and the plugin's host guard run; only the socket is replaced. + """ + holder = {"calls": []} - monkeypatch.setattr("tools.client.requests.Session", factory) + def install(responder): + def handler(request: httpx.Request) -> httpx.Response: + kwargs = { + "params": dict(request.url.params), + "headers": dict(request.headers), + "json": json.loads(request.content) if request.content else None, + } + url = str(request.url.copy_with(query=None)) + holder["calls"].append({"method": request.method, "url": url, **kwargs}) + result = responder(request.method, url, kwargs) + if isinstance(result.payload, str): + return httpx.Response(result.status_code, text=result.payload, headers=result.headers) + return httpx.Response(result.status_code, json=result.payload, headers=result.headers) + + monkeypatch.setattr("tools.sdk.httpx.HTTPTransport", lambda **_: httpx.MockTransport(handler)) return holder return install @@ -70,7 +69,7 @@ def _text(messages): def test_credential_validation_uses_account_api_and_hides_the_token(session_factory): def responder(method, url, kwargs): assert kwargs["params"]["token"] == TOKEN - assert kwargs["params"]["days"] == 1 + assert kwargs["params"]["days"] == "1" return FakeResponse(200, {"plan": "FREE", "status": "active", "token": TOKEN, "email": "hidden@example.com"}) session_factory(responder) @@ -91,10 +90,10 @@ def responder(method, url, kwargs): def test_web_search_uses_bearer_auth_and_not_the_query_token(session_factory): def responder(method, url, kwargs): assert url.endswith("/api/v1/web_search") - assert kwargs["headers"]["Authorization"] == f"Bearer {TOKEN}" + assert kwargs["headers"]["authorization"] == f"Bearer {TOKEN}" assert "token" not in (kwargs.get("params") or {}) assert kwargs["params"]["text"] == "hypothyroidism" - assert kwargs["params"]["snippetLength"] == "SHORT" + assert "snippetLength" not in kwargs["params"] # API default applies return FakeResponse( 200, { @@ -173,8 +172,8 @@ def responder(method, url, kwargs): def test_organization_search_drops_unexpected_contact_fields(session_factory): def responder(method, url, kwargs): assert "kg.diffbot.com" in url - assert "type:Organization" in kwargs["params"]["query"] - assert 'name:"Diffbot"' in kwargs["params"]["query"] + assert "type:Organization" in kwargs["json"]["query"] + assert 'name:"Diffbot"' in kwargs["json"]["query"] return FakeResponse( 200, { @@ -208,7 +207,7 @@ def responder(method, url, kwargs): def test_people_search_keeps_employment_and_drops_email(session_factory): def responder(method, url, kwargs): - assert "type:Person" in kwargs["params"]["query"] + assert "type:Person" in kwargs["json"]["query"] return FakeResponse( 200, { @@ -241,7 +240,7 @@ def test_news_search_passes_the_built_query(session_factory): seen = {} def responder(method, url, kwargs): - seen["query"] = kwargs["params"]["query"] + seen["query"] = kwargs["json"]["query"] return FakeResponse( 200, { @@ -271,43 +270,6 @@ def responder(method, url, kwargs): assert "text" not in payload["results"][0] -def test_enhance_sends_partial_identifiers(session_factory): - def responder(method, url, kwargs): - assert kwargs["params"]["type"] == "Organization" - assert kwargs["params"]["name"] == "Diffbot" - assert "email" not in kwargs["params"] - return FakeResponse( - 200, - { - "hits": 1, - "data": [ - { - "score": 0.7, - "entity": { - "name": "Diffbot", - "homepageUri": "https://www.diffbot.com", - "diffbotUri": "http://diffbot.com/entity/org1", - "allEmail": ["secret@example.com"], - }, - } - ], - }, - ) - - session_factory(responder) - tool = EnhanceEntityTool.from_credentials({"diffbot_token": TOKEN}) - payload = _json_message(list(tool.invoke({"entity_type": "Organization", "name": "Diffbot"}))) - assert payload["matches"][0]["score"] == 0.7 - assert payload["matches"][0]["entity"]["id"] == "org1" - assert "secret@example.com" not in json.dumps(payload) - - -def test_enhance_rejects_employer_for_an_organization(): - tool = EnhanceEntityTool.from_credentials({"diffbot_token": TOKEN}) - with pytest.raises(DiffbotError, match="employer"): - list(tool.invoke({"entity_type": "Organization", "name": "Diffbot", "employer": "Nobody"})) - - def test_resolve_entities_builds_a_dql_filter(session_factory): def responder(method, url, kwargs): assert method == "POST" @@ -345,7 +307,7 @@ def responder(method, url, kwargs): def test_timeout_becomes_a_safe_error(session_factory): def responder(method, url, kwargs): - raise requests.Timeout("timed out with token " + TOKEN) + raise httpx.ReadTimeout("timed out with token " + TOKEN) session_factory(responder) tool = SearchWebTool.from_credentials({"diffbot_token": TOKEN})