Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 30 additions & 9 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ name: Release
# tests, build, `pnpm pack`) and nothing else. Every step that publishes to npm
# or creates a GitHub release is gated on a real tag push, so a dispatch — on
# any branch, by anyone with write access — can never ship a version.
#
# npm auth is trusted publishing (OIDC). No NPM_TOKEN and no GitHub environment.
# The package must already exist on npm before a trusted publisher can be added.
on:
push:
tags: ["v*"]
Expand Down Expand Up @@ -93,22 +96,35 @@ jobs:
registry-url: https://registry.npmjs.org
- run: pnpm install --frozen-lockfile
- run: pnpm build
# Trusted publishing needs npm >= 11.5.1; Node 22 ships older. Use npm,
# not pnpm, so the OIDC token exchange works. No token is set.
- name: Update npm
run: |
npm install -g npm@latest
npm --version

# The dist-tag is derived from the version, not from the tag name: a
# prerelease (anything containing a `-`, e.g. 0.1.0-rc.1) publishes as
# `next` so a plain `npm install @diffbot/ai-sdk` keeps resolving to the
# last stable release. The guards above already proved the git tag equals
# this version.
- name: pnpm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Publish to npm
run: |
set -euo pipefail
name="$(node -p "require('./package.json').name")"
version="$(node -p "require('./package.json').version")"
case "$version" in
*-*) dist_tag=next ;;
*) dist_tag=latest ;;
esac
echo "Publishing $version with --tag $dist_tag."
pnpm publish --access public --provenance --no-git-checks --tag "$dist_tag"
if npm view "${name}@${version}" version >/dev/null 2>&1; then
echo "${name}@${version} is already on npm; skipping publish."
exit 0
fi
echo "Publishing $version with --tag $dist_tag (npm trusted publishing)."
# --ignore-scripts: dist/ was built above. No NODE_AUTH_TOKEN; npm
# authenticates with the trusted publisher on npmjs.com.
npm publish --access public --provenance --ignore-scripts --tag "$dist_tag"

github-release:
name: create GitHub release
Expand All @@ -125,7 +141,12 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
run: |
gh release create "$GITHUB_REF_NAME" \
--repo "$GITHUB_REPOSITORY" \
--title "$GITHUB_REF_NAME" \
--generate-notes
set -euo pipefail
if gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" >/dev/null 2>&1; then
echo "Release $GITHUB_REF_NAME already exists; leaving it."
else
gh release create "$GITHUB_REF_NAME" \
--repo "$GITHUB_REPOSITORY" \
--title "$GITHUB_REF_NAME" \
--generate-notes
fi
13 changes: 12 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Changelog

## 0.1.0 — Unreleased
## 0.1.0 - 2026-11-17

First API of `@diffbot/ai-sdk`: Diffbot tools, retrieval helpers, and an OpenAI-compatible provider
for the Vercel AI SDK (`ai` ^6 or ^7).
Expand All @@ -24,3 +24,14 @@ for the Vercel AI SDK (`ai` ^6 or ^7).
timeout.
- Examples: `examples/company-research` (tool-calling agent), `examples/manual-rag` (retrieve
helpers, no tool loop), and `examples/worker` (Workers `env` bag and streamed RAG).
- `extractOptions` on `diffbotExtract`, `loadExtract`, and `streamExtract` passes the SDK's
`ExtractOptions` (`timeout`, `mode`, `fields`, `fallback`, `discussion`, `paging`, …) through to
`extract`. The tool's model-facing input schema is unchanged, and `api`/`fmt` defaults stay.
- Opt-in `download` on `loadCrawl` / `streamCrawl`: start the job without watching, poll it, and
map the records from `crawlDownload` to documents (`defaultCrawlRecordMapper`). The default watch
stream is unchanged.

### Changed

- Depends on `@diffbot/typescript` `^0.6.0` (cumulative 0.4.0–0.6.0). DQL is POST with a JSON body; the token stays the query parameter. Unit-test mocks already assert that body. The dev-only `.pnpmfile.cjs` sibling link is removed. Refresh `pnpm-lock.yaml` from the registry once 0.6.0 is on npm; do not commit a `link:` or `file:` override.
- `retrieveKnowledgeGraph` takes `nonCanonicalFacts` as a named boolean. Passing that key through `extra` throws, matching the SDK.
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Precedence inside `execute` (lazy): injected `client` → `apiKey` → `resolveT

## Phase 4 scope (current)

Seven tools plus retrieval/loader helpers: `retrieveKnowledgeGraph`, `retrieveWebSearch`, `loadExtract`, `streamExtract`, `loadCrawl`, `streamCrawl`, `defaultCrawlMapper` (crawl is helpers-only, never a tool). Provider at `@diffbot/ai-sdk/provider` (`createDiffbot`). `diffbotAsk` defaults to `timeout: 120_000` when this package builds the client; injected clients keep their own timeout. `diffbotTools()` still builds one shared client, so that default covers the whole bundle when `diffbot_ask` is included. No examples yet (Phase 5). Live coverage is complete: every tool, both retrievers, `loadExtract`, and the provider (`generateText` + `streamText`).
Seven tools plus retrieval/loader helpers: `retrieveKnowledgeGraph`, `retrieveWebSearch`, `loadExtract`, `streamExtract`, `loadCrawl`, `streamCrawl`, `defaultCrawlMapper`, `defaultCrawlRecordMapper` (crawl is helpers-only, never a tool; `download` opts into start → poll → `crawlDownload`). Provider at `@diffbot/ai-sdk/provider` (`createDiffbot`). `diffbotAsk` defaults to `timeout: 120_000` when this package builds the client; injected clients keep their own timeout. `diffbotTools()` still builds one shared client, so that default covers the whole bundle when `diffbot_ask` is included. No examples yet (Phase 5). Live coverage is complete: every tool, both retrievers, `loadExtract`, and the provider (`generateText` + `streamText`).

## CI and live tests

Expand All @@ -22,7 +22,7 @@ Four workflows:
- `ci.yml` — unit-only, **no secrets** (lint, typecheck, unit tests + build on Node 20/22/24, the `ai@6`/`ai@7` peer matrix, empty-token live-skip guard, Workers bundle check). Keep it secret-free so it can run on fork PRs.
- `integration.yml` — `pnpm test:live`, on push to `main`, a twice-monthly drift schedule (1st and 15th at 14:17 UTC), and `workflow_dispatch`. Never `pull_request`/`pull_request_target`, so `DIFFBOT_API_TOKEN` is never exposed to fork PR code. A guard step fails the run on a missing or blank secret rather than skipping everything green.
- `canary.yml` — weekly `ai@latest` + `@ai-sdk/openai-compatible@latest` typecheck + unit tests. No secrets.
- `release.yml` — `v*` tag only; `workflow_dispatch` is a build-only rehearsal and every publish/release step is gated on the tag push. It fails early unless the tag equals `package.json`'s version and `"private"` is no longer `true` (Phase 6 flips that flag — do not change it before then).
- `release.yml` — `v*` tag only; `workflow_dispatch` is a build-only rehearsal and every publish/release step is gated on the tag push. It fails early unless the tag equals `package.json`'s version and `"private"` is not `true`. Publish uses npm trusted publishing (OIDC). There is no `NPM_TOKEN`. `@diffbot/ai-sdk` is not on npm yet, so a trusted publisher cannot be added until a one-time manual bootstrap publish creates the package.

`pnpm test:workers` dry-run bundles `tests/fixtures/worker/` for workerd and then scans the bundle: `wrangler deploy --dry-run` alone leaves static `node:*` imports in the output as externals, so the scan is what actually enforces Workers-cleanliness.

Expand Down
21 changes: 14 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,25 @@ After `pnpm test:peer:ai6` (or `:ai7`, or `:latest`) restore the lockfile — th
git checkout package.json pnpm-lock.yaml && pnpm install --frozen-lockfile
```

## Local SDK link (temporary)

The release dependency is `@diffbot/typescript` `^0.6.0`. Do not commit a `file:` / `link:` specifier
or a `.pnpmfile.cjs`. Until 0.6.0 is on npm, `pnpm-lock.yaml` still points at a sibling link and
cannot be refreshed from the registry. For a local unit-test run, link a built 0.6.0 checkout and
restore `pnpm-lock.yaml` before committing.

## Workflows

| Workflow | Triggers | Secrets |
| ----------------- | ------------------------------------------------- | ------------------------- |
| `ci.yml` | push to `main`, every `pull_request`, dispatch | **none** |
| `integration.yml` | push to `main`, 1st + 15th at 14:17 UTC, dispatch | `DIFFBOT_API_TOKEN` |
| `canary.yml` | Mondays at 14:17 UTC, dispatch | **none** |
| `release.yml` | `v*` tag; dispatch is a build-only rehearsal | `NPM_TOKEN` (publish job) |
| Workflow | Triggers | Secrets |
| ----------------- | ------------------------------------------------- | --------------------------------- |
| `ci.yml` | push to `main`, every `pull_request`, dispatch | **none** |
| `integration.yml` | push to `main`, 1st + 15th at 14:17 UTC, dispatch | `DIFFBOT_API_TOKEN` |
| `canary.yml` | Mondays at 14:17 UTC, dispatch | **none** |
| `release.yml` | `v*` tag; dispatch is a build-only rehearsal | **none** (npm trusted publishing) |

`ci.yml` runs lint + typecheck, unit tests + build on Node 20 / 22 / 24, the peer matrix (`ai@6` + `@ai-sdk/openai-compatible@2`, `ai@7` + `@3`), the empty-token live-skip guard, and the Workers bundle check. **It must stay secret-free** — that is what lets every job run on fork PRs.

`release.yml` publishes only from a `v*` tag push: it checks the tag equals `package.json`'s `version` and that `"private"` is no longer `true`, runs lint/typecheck/tests/build, then `pnpm publish --access public --provenance`, then creates the GitHub release. `workflow_dispatch` stops after `pnpm pack`; every publishing step is gated on the tag push, so a dispatch cannot ship a version. To release: bump `version`, commit, tag `vX.Y.Z`, push the tag.
`release.yml` publishes only from a `v*` tag push: it checks the tag equals `package.json`'s `version` and that `"private"` is not `true`, runs lint/typecheck/tests/build, then `npm publish --access public --provenance` via trusted publishing (OIDC, no `NPM_TOKEN`), then creates the GitHub release. `workflow_dispatch` stops after `pnpm pack`; every publishing step is gated on the tag push, so a dispatch cannot ship a version. To release: bump `version`, commit, tag `vX.Y.Z`, push the tag. npm cannot attach a trusted publisher until the package exists; the first publish of `@diffbot/ai-sdk` needs a one-time manual bootstrap (see the release report) before that tag will succeed.

## Live tests

Expand Down
17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -527,19 +527,26 @@ Each returns (or yields) `DiffbotDocument` objects:
`{ content: string; metadata: Record<string, unknown> }`. Options beyond the auth ones:

- `retrieveKnowledgeGraph` — `query` (DQL, required), `k` (default `10`), `from`, `filter`,
`exportspec`, `extra`, `format` (default `json`; other formats throw, since only JSON can be
shaped), plus `fields` / `contentFields` (default `DEFAULT_KG_CONTENT_FIELDS`) / `documentMapper`.
`exportspec`, `nonCanonicalFacts`, `extra` (only keys that are not named DQL options; 0.6.0
throws if a named key such as `nonCanonicalFacts` is passed through `extra`), `format` (default
`json`; other formats throw, since only JSON can be shaped), plus `fields` / `contentFields`
(default `DEFAULT_KG_CONTENT_FIELDS`) / `documentMapper`.
- `retrieveWebSearch` — `query` (required), `k` (default `10`), `maxTokens`, `snippetLength`, plus
`fields` / `contentFields` (default `DEFAULT_WEB_CONTENT_FIELDS`) / `documentMapper`.
- `loadExtract` / `streamExtract` — `urls` (required), `api` (default `analyze`), `fmt` (default
`markdown`), `documentMapper(url, raw)`. `metadata` carries `url`, `title`, `pageUrl`,
`markdown`), `extractOptions` (the SDK's `ExtractOptions`: `timeout`, `mode`, `fields`, `fallback`,
`discussion`, `paging`, `extra`, …), `documentMapper(url, raw)`. `metadata` carries `url`, `title`, `pageUrl`,
`resolvedPageUrl` and `type`. URLs are fetched one at a time, in order, and an extraction failure
throws (as in `@diffbot/langchain`): `loadExtract` rejects, while `streamExtract` has already
yielded the documents before the failing URL.
- `loadCrawl` / `streamCrawl` — `site` (required), `crawlOptions` (the SDK's `CrawlOptions`; `watch`
defaults to `true` so URL events are emitted), `eventMapper(event)` returning a document or `null`
to skip. The default `content` is the crawled URL itself, with `url`, `status` and
`crawl_timestamp` in `metadata`.
`crawl_timestamp` in `metadata`. Pass `download: true` (or `{ num, type, recordMapper }`) to skip
the watch stream: the job is started without `watch`, polled until it finishes, and its extracted
records are fetched with the SDK's `crawlDownload`, one document per record (`content` is the
record's `content`, falling back to its title; `eventMapper` does not apply). `type: "urls"` is
rejected, since that report is CSV.

`defaultCrawlMapper` drops the `job_created` event, so with `watch: false` you get no documents at
all — that combination only makes sense with an `eventMapper`. Pass `crawlOptions.jobName` if you
Expand Down Expand Up @@ -647,7 +654,7 @@ Every runtime export of `@diffbot/ai-sdk`:
| `diffbotWebSearch`, `diffbotExtract`, `diffbotKnowledgeGraph`, `diffbotEntities`, `diffbotOntology`, `diffbotDqlProbe`, `diffbotAsk` | Model-facing `tool()` |
| `diffbotTools` | Bundle |
| `retrieveKnowledgeGraph`, `retrieveWebSearch` | Retrieval |
| `loadExtract`, `streamExtract`, `loadCrawl`, `streamCrawl`, `defaultCrawlMapper` | Loaders |
| `loadExtract`, `streamExtract`, `loadCrawl`, `streamCrawl`, `defaultCrawlMapper`, `defaultCrawlRecordMapper` | Loaders |
| `dictToDocument`, `shapeExtractResponse`, `DEFAULT_KG_CONTENT_FIELDS`, `DEFAULT_WEB_CONTENT_FIELDS` | Shaping |
| `CrawlEventType` | Re-export from the SDK |

Expand Down
6 changes: 3 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"name": "@diffbot/ai-sdk",
"version": "0.1.0-dev.0",
"version": "0.1.0",
"description": "Diffbot tools and LLM provider for the Vercel AI SDK — web search, extract, DQL, ontology, and RAG",
"type": "module",
"private": true,
"private": false,
"packageManager": "pnpm@10.33.0",
"main": "./dist/index.cjs",
"module": "./dist/index.js",
Expand Down Expand Up @@ -61,7 +61,7 @@
"test:workers": "tests/fixtures/worker/dry-run.sh"
},
"dependencies": {
"@diffbot/typescript": "^0.3.0"
"@diffbot/typescript": "^0.6.0"
},
"peerDependencies": {
"@ai-sdk/openai-compatible": "^2.0.0 || ^3.0.0",
Expand Down
10 changes: 5 additions & 5 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,10 @@ export {
loadCrawl,
streamCrawl,
defaultCrawlMapper,
defaultCrawlRecordMapper,
type CrawlDownloadLoadOptions,
type CrawlEventMapper,
type CrawlRecordMapper,
type StreamCrawlOptions,
} from "./retrieve/crawl.js";

Expand Down
Loading
Loading