Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,22 @@ on:
tags: ["v*"]
workflow_dispatch:

permissions:
contents: write
# Least privilege by default; the release job elevates its own permissions.
permissions: {}

# Re-pushing a tag (or re-dispatching) cancels an in-flight run for the same
# ref so a stuck build can't pile up behind a newer attempt.
# Never cancel an in-flight run: cancelling a publish midway could leave PyPI
# and the GitHub Release out of sync. Runs for the same ref queue instead.
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: false

jobs:
build:
name: Build distribution
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -58,6 +60,12 @@ jobs:
runs-on: ubuntu-latest
# Only publish for real tag pushes; workflow_dispatch runs just build + verify.
if: startsWith(github.ref, 'refs/tags/')
# No GitHub environment: the PyPI trusted publisher has none configured.
permissions:
# Required for PyPI trusted publishing (OIDC).
id-token: write
# Required to create the GitHub Release.
contents: write
steps:
- name: Download artifacts
uses: actions/download-artifact@v4
Expand All @@ -77,9 +85,7 @@ jobs:
python-version: "3.12"

- name: Publish to PyPI
env:
UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}
run: uv publish
run: uv publish --trusted-publishing always

- name: Publish GitHub Release
env:
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Changelog

## 0.3.0 - 2026-11-17

Require `diffbot>=3.4.0,<4`. 0.2.0 is already on PyPI, so this release is 0.3.0.

DQL follows the SDK: POST `https://kg.diffbot.com/kg/v3/dql` with a JSON body; the token stays the `token` query parameter. GET remains the low-level escape hatch.

Relock with `uv lock --upgrade-package diffbot` once 3.4.0 is on PyPI. That command needs the sibling `../langchain` checkout so the editable `langchain-core` / `langchain-tests` sources stay in `uv.lock`. `make dev-sdk` is only a pre-publish local install; do not commit it.

Publishing is PyPI trusted publishing from `.github/workflows/release.yml` (no token, no GitHub environment). Tag `v0.3.0`. The SDK still has no `py.typed` marker, so the mypy override stays.
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ LangChain integration package for the Diffbot APIs (Knowledge Graph, Extract, We
## Stack

- **Build/deps**: `uv` (not Poetry/pip). `pyproject.toml` is PEP 621 with `hatchling` as the build backend. Dependency groups (`test`, `lint`, `typing`) are declared under `[dependency-groups]` and invoked via `uv run --group <name> ...`.
- **HTTP / Diffbot transport**: the official [`diffbot-python`](https://github.com/diffbot/diffbot-python) SDK. It wraps `httpx` under the hood, so `respx` continues to work for unit tests — mocks target the real upstream URLs (KG: `https://kg.diffbot.com/kg/v3/dql`, web search: `https://llm.diffbot.com/api/v1/web_search`, extract: `https://api.diffbot.com/v3/analyze`, ask: `https://llm.diffbot.com/rag/v1/chat/completions`, NLP: `https://nl.diffbot.com/v1/`, crawl: `https://api.diffbot.com/v3/crawl`, ontology: `https://kg.diffbot.com/kg/ontology`). `diffbot-python` is published on PyPI and resolved from there as a normal dependency (`diffbot-python>=0.1.0`); `langchain-core` / `langchain-tests` are still resolved during local dev from the sibling `../langchain/` checkout via `[tool.uv.sources]` — release builds use the published versions.
- **HTTP / Diffbot transport**: the official [`diffbot-python`](https://github.com/diffbot/diffbot-python) SDK. It wraps `httpx` under the hood, so `respx` continues to work for unit tests — mocks target the real upstream URLs (KG: `POST https://kg.diffbot.com/kg/v3/dql` with a JSON body, web search: `https://llm.diffbot.com/api/v1/web_search`, extract: `https://api.diffbot.com/v3/analyze`, ask: `https://llm.diffbot.com/rag/v1/chat/completions`, NLP: `https://nl.diffbot.com/v1/`, crawl: `https://api.diffbot.com/v3/crawl`, ontology: `https://kg.diffbot.com/kg/ontology`). It is published on PyPI as `diffbot` and declared as `diffbot>=3.4.0,<4`. Until 3.4.0 is on PyPI, `make dev-sdk` can install a sibling checkout over the synced env; run the `make` targets with `UV_NO_SYNC=true` so `uv run` doesn't re-resolve and replace it. Do not commit that editable install. Relock with `uv lock --upgrade-package diffbot` once 3.4.0 is published (that command needs the sibling `../langchain` checkout). `langchain-core` / `langchain-tests` are still resolved during local dev from the sibling `../langchain/` checkout via `[tool.uv.sources]` — release builds use the published versions.
- **Models**: `pydantic` v2.
- **LangChain**: targets `langchain-core >=1.0,<2.0`. During local dev `langchain-core` and `langchain-tests` are resolved from a sibling `../langchain/` checkout via `[tool.uv.sources]` — release builds use the published versions.
- **Test runner**: `pytest` with `asyncio_mode = "auto"`.
Expand Down
10 changes: 9 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
.PHONY: format lint test test_integration typing build clean bump-patch bump-minor bump-major set-token-pypi set-token-testpypi release-test release verify-release-test verify-release
.PHONY: dev-sdk format lint test test_integration typing build clean bump-patch bump-minor bump-major set-token-pypi set-token-testpypi release-test release verify-release-test verify-release

# Dev-only: install a local Diffbot SDK checkout over the synced env (no-op if
# ../diffbot-python is absent). The release dependency is diffbot>=3.4.0,<4 from
# PyPI. Pair with UV_NO_SYNC=true so `make test` doesn't re-resolve and replace it.
# Do not commit the editable install.
dev-sdk:
@if [ -d ../diffbot-python ]; then uv pip install -e ../diffbot-python; \
else echo "../diffbot-python not found; skipping"; fi

format:
uv run --group lint ruff format .
Expand Down
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,8 @@ for doc in loader.lazy_load():

`DiffbotExtractTool` returns a structured `{"error": ..., "errorCode": ...}` dict when Diffbot reports an extraction failure (200 with `errorCode`), so agents can react and try another URL instead of catching an exception. Auth / rate-limit errors propagate as `diffbot.errors.AuthError` / `RateLimitError`.

Both `DiffbotExtractTool` and `DiffbotExtractLoader` take an `extract_options` dict that is forwarded to `Diffbot.extract` on every call: `timeout`, `mode`, `fields`, `fallback`, `discussion`, `paging`, `max_pages`, `natural_language`, `summary_num_sentences`, `max_tags`, `tag_confidence`, `category_confidence`, `use_proxy`, and `extra` (for `proxy` / `proxyAuth`). It is empty by default. Requires `diffbot>=3.4.0,<4`.

## Crawl a site

`DiffbotCrawlLoader` drives a Diffbot crawl job and yields one `Document` per crawled URL. The `page_content` is the URL itself (the crawl API surfaces URLs, not page contents) — chain it with `DiffbotExtractLoader` to fetch the content of each URL.
Expand All @@ -139,6 +141,8 @@ for doc in loader.lazy_load():
print(doc.metadata["url"], doc.metadata["status"])
```

By default the loader watches the job. Pass `download=True` for an opt-in start / poll / download path: it starts a job (or reuses `job_name`), polls until the job finishes (`poll_interval`, `poll_timeout`), then calls `crawl_download` and yields one `Document` per extracted record with real page content. `download_type`, `download_num`, and `record_mapper` tune the download.

## ChatDiffbot

```python
Expand Down
177 changes: 172 additions & 5 deletions langchain_diffbot/document_loaders.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,16 @@

from __future__ import annotations

import asyncio
import time
from collections.abc import AsyncIterator, Callable, Iterator
from typing import Any

from diffbot.crawl import CrawlEvent, CrawlEventType
from diffbot.errors import JobError
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document
from pydantic import Field
from pydantic import Field, field_validator

from langchain_diffbot._base import _BaseDiffbotComponent

Expand All @@ -33,6 +36,40 @@ def _default_extract_mapper(url: str, raw: dict[str, Any]) -> Document:
return Document(page_content=page_content, metadata=metadata)


CrawlRecordMapper = Callable[[dict[str, Any]], Document | None]
"""`(record) -> Document | None` for `crawl_download` records. None skips."""

# Crawler job status codes that mean "still running" (matches the SDK's own
# watch loop); anything else is terminal.
_CRAWL_RUNNING_STATUSES = (0, 7)


def _default_record_mapper(record: dict[str, Any]) -> Document | None:
url = record.get("pageUrl") or record.get("resolvedPageUrl") or ""
return _default_extract_mapper(url, {"objects": [record]})


def _check_job_done(job_name: str, job: dict[str, Any]) -> bool:
"""Return True when the job is terminal; raise `JobError` if it failed.

`crawl_get_job` returns `{}` for an unknown job; that is an error, not a
job that is "still running" (otherwise we'd poll until `poll_timeout`).
"""
if not job:
raise JobError(job_name, f"Crawler job {job_name!r} not found")
status = job.get("jobStatus") or {}
if status.get("status", 0) in _CRAWL_RUNNING_STATUSES:
return False
message = status.get("message", "")
if "fail" in message.lower() or "error" in message.lower():
raise JobError(job_name, f"Crawler job failed: {message}")
return True


def _poll_timeout_error(job_name: str, timeout: float) -> TimeoutError:
return TimeoutError(f"Crawl job {job_name!r} not finished after {timeout}s")


def _default_crawl_mapper(event: CrawlEvent) -> Document | None:
# Non-URL events (e.g. JOB_CREATED) are skipped by default. The Document's
# page_content is the URL itself — the crawl SDK only surfaces URLs, not
Expand Down Expand Up @@ -74,6 +111,15 @@ class DiffbotExtractLoader(_BaseDiffbotComponent, BaseLoader):
fmt: str = "markdown"
"""Output format. `markdown` uses Diffbot's LLM-optimized mode."""

extract_options: dict[str, Any] = Field(default_factory=dict)
"""Keyword options forwarded to `Diffbot.extract` for every URL.

Any of the SDK's extract options (`timeout`, `mode`, `fields`, `fallback`,
`discussion`, `paging`, `max_pages`, `natural_language`,
`summary_num_sentences`, `max_tags`, `tag_confidence`,
`category_confidence`, `use_proxy`, `extra`). Empty by default.
"""

document_mapper: ExtractDocumentMapper | None = None
"""Optional `(url, raw_response) -> Document` override."""

Expand All @@ -85,14 +131,18 @@ def lazy_load(self) -> Iterator[Document]:
"""Yield one `Document` per URL, calling `extract` synchronously."""
with self._sync_db() as db:
for url in self.urls:
raw = db.extract(url, api=self.api, fmt=self.fmt)
raw = db.extract(
url, api=self.api, fmt=self.fmt, **self.extract_options
)
yield self._to_doc(url, raw)

async def alazy_load(self) -> AsyncIterator[Document]:
"""Yield one `Document` per URL, calling `extract` asynchronously."""
async with self._async_db() as db:
for url in self.urls:
raw = await db.extract(url, api=self.api, fmt=self.fmt)
raw = await db.extract(
url, api=self.api, fmt=self.fmt, **self.extract_options
)
yield self._to_doc(url, raw)


Expand Down Expand Up @@ -120,6 +170,52 @@ class DiffbotCrawlLoader(_BaseDiffbotComponent, BaseLoader):
event_mapper: CrawlEventMapper | None = None
"""Optional `(event) -> Document | None` override. Return None to skip."""

download: bool = False
"""Opt in to the start / poll / download path instead of watching.

When True the loader starts a job (or reuses `job_name`), polls
`crawl_get_job` until it finishes, then calls `crawl_download` and yields a
`Document` per extracted record with real page content. `watch` and
`event_mapper` are not used in this mode. Default False keeps the
URL-per-event watch behavior.
"""

job_name: str | None = None
"""Existing crawl job to poll and download (only with `download=True`).

If unset, a new job is started for `site` (using `crawl_kwargs`, with
`watch=False`; set `job_name` inside `crawl_kwargs` to choose its name).
"""

poll_interval: float = 2.0
"""Seconds between job-status polls (only with `download=True`)."""

poll_timeout: float | None = 600.0
"""Max seconds to wait for the job to finish; None waits forever."""

download_type: str | None = None
"""Report passed to `crawl_download(type=...)`.

`"urls"` is rejected: that report is CSV text, not records. Use the default
watch mode (`download=False`) for a URL-per-`Document` crawl.
"""

download_num: int | None = None
"""Cap on records passed to `crawl_download(num=...)`."""

record_mapper: CrawlRecordMapper | None = None
"""Optional `(record) -> Document | None` override for downloaded records."""

@field_validator("download_type")
@classmethod
def _reject_urls_report(cls, v: str | None) -> str | None:
if v == "urls":
raise ValueError(
'download_type="urls" returns CSV text, not records; use the '
"default watch mode (download=False) to get one Document per URL"
)
return v

def _kwargs(self) -> dict[str, Any]:
kw = dict(self.crawl_kwargs)
kw.setdefault("watch", True)
Expand All @@ -129,18 +225,89 @@ def _map_event(self, event: CrawlEvent) -> Document | None:
mapper = self.event_mapper or _default_crawl_mapper
return mapper(event)

def _records_to_docs(self, records: list[dict[str, Any]]) -> Iterator[Document]:
mapper = self.record_mapper or _default_record_mapper
for record in records:
doc = mapper(record)
if doc is not None:
yield doc

def _deadline(self) -> float | None:
if self.poll_timeout is None:
return None
return time.monotonic() + self.poll_timeout

def _start_kwargs(self) -> dict[str, Any]:
kw = dict(self.crawl_kwargs)
kw["watch"] = False
return kw

def lazy_load(self) -> Iterator[Document]:
"""Drive a crawl synchronously and yield a `Document` per mapped event."""
"""Drive a crawl synchronously and yield a `Document` per mapped event.

With `download=True`, start (or reuse) a job, poll until it finishes,
and yield a `Document` per downloaded record instead.
"""
if self.download:
yield from self._download_sync()
return
with self._sync_db() as db:
for event in db.crawl(self.site, **self._kwargs()):
doc = self._map_event(event)
if doc is not None:
yield doc

async def alazy_load(self) -> AsyncIterator[Document]:
"""Drive a crawl asynchronously and yield a `Document` per mapped event."""
"""Drive a crawl asynchronously and yield a `Document` per mapped event.

With `download=True`, start (or reuse) a job, poll until it finishes,
and yield a `Document` per downloaded record instead.
"""
if self.download:
async for downloaded in self._download_async():
yield downloaded
return
async with self._async_db() as db:
async for event in db.crawl(self.site, **self._kwargs()):
doc = self._map_event(event)
if doc is not None:
yield doc

def _download_sync(self) -> Iterator[Document]:
with self._sync_db() as db:
name = self.job_name
if name is None:
for event in db.crawl(self.site, **self._start_kwargs()):
if event.event_type == CrawlEventType.JOB_CREATED:
name = event.details["job_name"]
if name is None:
raise RuntimeError("Diffbot crawl did not report a job name")
deadline = self._deadline()
while not _check_job_done(name, db.crawl_get_job(name)):
if deadline is not None and time.monotonic() >= deadline:
raise _poll_timeout_error(name, self.poll_timeout or 0)
time.sleep(self.poll_interval)
records = db.crawl_download(
name, format="json", type=self.download_type, num=self.download_num
)
yield from self._records_to_docs(records)

async def _download_async(self) -> AsyncIterator[Document]:
async with self._async_db() as db:
name = self.job_name
if name is None:
async for event in db.crawl(self.site, **self._start_kwargs()):
if event.event_type == CrawlEventType.JOB_CREATED:
name = event.details["job_name"]
if name is None:
raise RuntimeError("Diffbot crawl did not report a job name")
deadline = self._deadline()
while not _check_job_done(name, await db.crawl_get_job(name)):
if deadline is not None and time.monotonic() >= deadline:
raise _poll_timeout_error(name, self.poll_timeout or 0)
await asyncio.sleep(self.poll_interval)
records = await db.crawl_download(
name, format="json", type=self.download_type, num=self.download_num
)
for rec_doc in self._records_to_docs(records):
yield rec_doc
14 changes: 12 additions & 2 deletions langchain_diffbot/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,16 @@ class DiffbotExtractTool(_BaseDiffbotComponent, BaseTool):
)
args_schema: type[BaseModel] = _DiffbotExtractInput

extract_options: dict[str, Any] = Field(default_factory=dict)
"""Keyword options forwarded to `Diffbot.extract` on every call.

Any of the SDK's extract options (`timeout`, `mode`, `fields`, `fallback`,
`discussion`, `paging`, `max_pages`, `natural_language`,
`summary_num_sentences`, `max_tags`, `tag_confidence`,
`category_confidence`, `use_proxy`, `extra`). Empty by default. Set by the
developer, not the agent.
"""

@staticmethod
def _shape_response(raw: dict[str, Any]) -> dict[str, Any]:
objects = raw.get("objects") or []
Expand All @@ -74,7 +84,7 @@ def _run(
) -> dict[str, Any]:
try:
with self._sync_db() as db:
raw = db.extract(url, api=api, fmt=fmt)
raw = db.extract(url, api=api, fmt=fmt, **self.extract_options)
except ExtractionError as e:
return {"error": str(e), "errorCode": e.error_code}
return self._shape_response(raw)
Expand All @@ -88,7 +98,7 @@ async def _arun(
) -> dict[str, Any]:
try:
async with self._async_db() as db:
raw = await db.extract(url, api=api, fmt=fmt)
raw = await db.extract(url, api=api, fmt=fmt, **self.extract_options)
except ExtractionError as e:
return {"error": str(e), "errorCode": e.error_code}
return self._shape_response(raw)
Expand Down
Loading
Loading