Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 15 additions & 8 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ on:
tags: ["v*"]
workflow_dispatch:

# Re-pushing a tag (or re-dispatching) cancels an in-flight run for the same
# ref so a stuck build can't pile up behind a newer attempt.
# Never cancel an in-flight run: cancelling a publish midway could leave npm
# and the GitHub Release out of sync. Runs for the same ref queue instead.
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: true
cancel-in-progress: false

jobs:
build:
Expand Down Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Build
run: pnpm build

# Bundle everything `pnpm publish` needs (package.json, README, LICENSE,
# Bundle everything `npm publish` needs (package.json, README, LICENSE,
# dist/) so the publish job below can run without checking out the repo.
- name: Assemble publishable package
run: |
Expand Down Expand Up @@ -83,17 +83,24 @@ jobs:
- name: List artifact contents
run: ls -lR package

- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 22
registry-url: "https://registry.npmjs.org"

# npm trusted publishing (OIDC) needs npm >= 11.5.1; Node 22 ships older.
- name: Update npm
run: |
npm install -g npm@latest
npm --version

# Uses npm (not pnpm) so the OIDC token exchange works. No token is set:
# npm authenticates via the trusted publisher configured on npmjs.com.
# --ignore-scripts: the package was already built in the build job, and
# package.json's prepublishOnly calls pnpm/tsup, absent in this job.
- name: Publish to npm
working-directory: package
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
run: pnpm publish --access public --no-git-checks --provenance
run: npm publish --access public --provenance --ignore-scripts

- name: Publish GitHub Release
env:
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Changelog

## 0.2.0 - 2026-11-17

Ship the unpublished 0.2.0 line against `@diffbot/typescript` ^0.6.0.

Peer, devDependency, `examples/company-research`, `examples/dql-explorer`, and `fixtures/worker` move off ^0.5.0 and ^0.2.0. DQL tests already assert POST with a JSON body; the token stays on the query string.

The dev-only `.pnpmfile.cjs` sibling link is removed. Refresh the root lock and the example/fixture locks after 0.6.0 is on npm (`pnpm install` / `pnpm update @diffbot/typescript`). Example `link:../..` entries for `@diffbot/langchain` itself stay local dev links.

Publishing is npm trusted publishing from `.github/workflows/release.yml` (no token, no GitHub environment). Tag `v0.2.0`.
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ src/
├── chat_models.ts # ChatDiffbot (wraps `ask`)
├── loaders/
│ ├── extract.ts # DiffbotExtractLoader
│ └── crawl.ts # DiffbotCrawlLoader, defaultCrawlMapper
│ └── crawl.ts # DiffbotCrawlLoader, defaultCrawlMapper, defaultCrawlRecordMapper
├── retrievers/
│ ├── knowledge-graph.ts # DiffbotKnowledgeGraphRetriever
│ └── web-search.ts # DiffbotWebSearchRetriever
Expand Down
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,8 @@ for await (const doc of loader.lazyLoad()) {

`DiffbotExtractTool` returns a structured `{ error, errorCode }` object when Diffbot reports an extraction failure (a 200 response carrying an `errorCode`), so an agent can react and try another URL instead of having to catch an exception. Auth and rate-limit failures still throw — those are infrastructure problems, not per-call signals.

Both accept `extractOptions`, the SDK's `ExtractOptions` (`timeout`, `mode`, `fields`, `fallback`, `discussion`, `paging`, `useProxy`, `extra`, ...), applied to every call. `api` / `fmt` keep their `analyze` / `markdown` defaults. On the tool, `extractOptions` is fixed at construction and is not part of the agent-facing call schema — the agent only chooses the URL.

`DiffbotExtractLoader` implements `lazyLoad()` as well as `load()`. Prefer `lazyLoad()` for long URL lists: it yields each `Document` as its request finishes, instead of buffering the whole batch.

## Crawl a site
Expand All @@ -175,6 +177,29 @@ for await (const doc of loader.lazyLoad()) {

The loader defaults `watch: true` into its `crawlOptions`, because without it the SDK emits a single `job_created` event and nothing else. Pass `crawlOptions: { watch: false, ... }` if you want to fire a job and walk away, and `eventMapper` to keep events the default mapper skips.

### Non-watch mode: `crawlDownload`

Pass `download` to skip the watch-style stream. The loader creates the job (always with `watch: false`), polls `crawlGetJob` until it finishes, then fetches the extracted records once with the SDK's `crawlDownload`. Each record becomes a `Document` whose `pageContent` is the record's extracted `content`. This suits runtimes that can't sit in a watch loop. It is opt-in; without `download` the loader behaves exactly as above.

```ts
import { DiffbotClient, resolveTokenFromEnv } from "@diffbot/typescript";
import { DiffbotCrawlLoader } from "@diffbot/langchain";

const client = new DiffbotClient({ token: resolveTokenFromEnv() });

/* Create a job, wait for it, download its records. */
const fresh = new DiffbotCrawlLoader({
client,
site: "https://www.diffbot.com",
download: { num: 50, pollInterval: 5000, maxWait: 600_000 },
});

/* Or download a job that already finished (no `site` needed). */
const existing = new DiffbotCrawlLoader({ client, download: { jobName: "my-finished-job" } });
```

`download` options: `jobName` (reuse an existing job instead of creating one), `num` (only the N most recently processed URLs), `pollInterval` (ms, default 2000), `maxWait` (ms, default 600000 like the Python loader's `poll_timeout`; `Infinity` waits indefinitely) and `recordMapper` (`(record) => Document | null`; the default is exported as `defaultCrawlRecordMapper` and reads `content`, falling back to `text`). A job that finishes with a failure message throws a `JobError`. The record's echoed `token` is never copied into metadata.

## ChatDiffbot

```ts
Expand Down
2 changes: 1 addition & 1 deletion examples/company-research/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
},
"dependencies": {
"@diffbot/langchain": "link:../..",
"@diffbot/typescript": "^0.2.0",
"@diffbot/typescript": "^0.6.0",
"@langchain/anthropic": "^1.5.4",
"@langchain/core": "^1.2.5",
"dotenv": "^17.2.3",
Expand Down
10 changes: 5 additions & 5 deletions examples/company-research/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/dql-explorer/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
},
"dependencies": {
"@diffbot/langchain": "link:../..",
"@diffbot/typescript": "^0.2.0",
"@diffbot/typescript": "^0.6.0",
"@hono/node-server": "^1.19.0",
"@langchain/anthropic": "^1.5.4",
"@langchain/core": "^1.2.5",
Expand Down
10 changes: 5 additions & 5 deletions examples/dql-explorer/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion fixtures/worker/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
},
"dependencies": {
"@diffbot/langchain": "link:../..",
"@diffbot/typescript": "^0.2.0",
"@diffbot/typescript": "^0.6.0",
"@langchain/core": "^1.2.5"
},
"devDependencies": {
Expand Down
10 changes: 5 additions & 5 deletions fixtures/worker/pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,11 @@
"zod": "^3.25.76 || ^4"
},
"peerDependencies": {
"@diffbot/typescript": "^0.2.0",
"@diffbot/typescript": "^0.6.0",
"@langchain/core": "^1.0.0"
},
"devDependencies": {
"@diffbot/typescript": "^0.2.0",
"@diffbot/typescript": "^0.6.0",
"@eslint/js": "^9.20.0",
"@langchain/core": "^1.2.5",
"@types/node": "^22.15.21",
Expand Down
10 changes: 5 additions & 5 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 7 additions & 4 deletions src/chat_models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
other knobs.
*/

import { ask, type ChatMessage, type DiffbotClient } from "@diffbot/typescript";
import { ask, type ChatMessage, type ChatRole, type DiffbotClient } from "@diffbot/typescript";
import type { CallbackManagerForLLMRun } from "@langchain/core/callbacks/manager";
import {
BaseChatModel,
Expand All @@ -24,7 +24,7 @@ import { assertClient, type DiffbotComponentFields } from "./base.js";

function toDiffbotMessages(messages: BaseMessage[]): ChatMessage[] {
return messages.map((message) => {
let role: string;
let role: ChatRole;
switch (message.getType()) {
case "human":
role = "user";
Expand All @@ -36,8 +36,11 @@ function toDiffbotMessages(messages: BaseMessage[]): ChatMessage[] {
role = "system";
break;
default:
/* Fall back on the message's own type for anything exotic. */
role = message.getType();
/*
The endpoint only knows system/user/assistant, so anything exotic
(tool, function, ...) is sent as a user turn.
*/
role = "user";
}
const content =
typeof message.content === "string" ? message.content : JSON.stringify(message.content);
Expand Down
3 changes: 3 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ export {
export {
DiffbotCrawlLoader,
defaultCrawlMapper,
defaultCrawlRecordMapper,
type CrawlEventMapper,
type CrawlRecordMapper,
type DiffbotCrawlDownloadOptions,
type DiffbotCrawlLoaderFields,
} from "./loaders/crawl.js";

Expand Down
Loading
Loading