Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
cc74046
feat(transport): endpoint registry, per-host bases, and full HTTP cap…
shixish Nov 13, 2026
4f8c4ab
fix(dql,crawl): stop corrupting binary exports; parse quoted CSV fields
shixish Nov 13, 2026
f92df1b
types: vendor Diffbot OpenAPI specs and generate src/generated/*
shixish Nov 13, 2026
58d34a2
types: retype every existing handler's response and give dql real ove…
shixish Nov 13, 2026
912ae27
test: add type tests (expectTypeOf/satisfies) and cover the new runti…
shixish Nov 13, 2026
8c50d23
lint: add typescript-eslint strict-type-checked on src/
shixish Nov 13, 2026
3368bc4
ci: run lint and type tests; add an Unreleased changelog entry for Ph…
shixish Nov 13, 2026
c4de121
docs: note two deliberately deferred Phase 0 items
shixish Nov 13, 2026
f7cf83c
feat(dql): send DQL as POST with a JSON body
shixish Nov 17, 2026
31c0ef2
ci: live tests on push to main, workflow_dispatch, cron 17 14 1,15 * *
shixish Nov 17, 2026
3bf56d1
docs: close Phase 0 leftovers under the reduced-scope plan; name 0.4.…
shixish Nov 17, 2026
676e623
fix(http): discard a retried attempt's response body
shixish Nov 18, 2026
62bc458
fix(crawl): throw APIError when create answers HTTP 200 with errorCode
shixish Nov 18, 2026
53b9ff7
feat(account): account({ days }) and verifyToken() over GET /v4/account
shixish Nov 17, 2026
73d1a1c
feat(crawl): public crawlDownload() over GET /v3/crawl/data; watcher …
shixish Nov 17, 2026
ba4014d
feat(extract): optional 5th arg ExtractOptions (timeout, mode, fields…
shixish Nov 17, 2026
2bcf39d
test(types): contract test tying named option keys to spec params
shixish Nov 17, 2026
f1fd59b
docs: S1 README snippets, 0.5.0 changelog, live cases, leftovers
shixish Nov 17, 2026
556f2ae
docs: S1 leftovers record DQL POST parity as confirmed for tested cases
shixish Nov 18, 2026
957d098
feat(dql): named params type, col, filterExclude, exportseparator, ex…
shixish Nov 17, 2026
731b071
feat(crawl): named create options and multi-value customHeaders
shixish Nov 17, 2026
42feee1
feat(nlp): nlp() on the existing NL endpoint, entities() unchanged
shixish Nov 17, 2026
e72463d
feat(web-search): accept 1–5 text values as repeated text=
shixish Nov 17, 2026
dcfb661
docs: S2 README snippets, 0.6.0 changelog, live cases, leftovers
shixish Nov 17, 2026
253208a
docs: record Q2/Q3 live confirmations in S2 leftovers
shixish Nov 18, 2026
dd0a256
refactor(crawl): spread CrawlOptions into buildCrawlParams instead of…
shixish Nov 18, 2026
43c078c
docs: S2 leftovers no longer call Q2 open; note crawl create errorCod…
shixish Nov 18, 2026
928f0bb
fix(http): stop retrying when Retry-After exceeds maxDelayMs
shixish Nov 18, 2026
c75d51c
chore(release): 0.6.0
shixish Nov 18, 2026
7b021ae
docs: one 0.6.0 changelog entry; drop the resolved leftovers files
shixish Nov 18, 2026
ebd638c
ci: add tag-triggered release workflow (npm trusted publishing + GitH…
shixish Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/live-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Live tests

# Hits the real Diffbot APIs with a repository secret. Deliberately NOT triggered by
# pull_request / pull_request_target (including forks); never add those triggers.
on:
push:
branches: [main]
workflow_dispatch:
schedule:
- cron: "17 14 1,15 * *"

permissions:
contents: read

jobs:
live:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
timeout-minutes: 20
concurrency:
group: live-tests
cancel-in-progress: false
env:
DIFFBOT_TOKEN: ${{ secrets.DIFFBOT_TOKEN }}
# The library's resolver reads DIFFBOT_API_TOKEN; same secret, never echoed.
DIFFBOT_API_TOKEN: ${{ secrets.DIFFBOT_TOKEN }}
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm test:live
220 changes: 220 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,220 @@
name: Release

# Publishes @diffbot/typescript to npm (with provenance) and creates the matching
# GitHub Release. Triggered by pushing a version tag (e.g. v0.6.0); also runnable
# manually, which runs every gate plus `npm publish --dry-run` and publishes nothing.
#
# npm auth: trusted publishing (OIDC) first. If no trusted publisher is configured on
# npmjs.com, npm falls back to the NPM_TOKEN repository secret (if set).
# Live tests are deliberately NOT run here; see live-tests.yml.
on:
push:
tags: ["v*"]
workflow_dispatch:

permissions:
contents: read

# Re-pushing a tag (or re-dispatching) queues behind an in-flight run for the same
# ref. Unlike a binaries build, a publish must never be cancelled halfway.
concurrency:
group: release-${{ github.ref }}
cancel-in-progress: false

jobs:
verify:
name: Verify version
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
version: ${{ steps.version.outputs.version }}
steps:
- uses: actions/checkout@v4

# Fail fast if package.json and src/constants.ts disagree, or the tag doesn't
# match them, so npm, the User-Agent (VERSION) and the GitHub Release can never
# report different versions.
- name: Check tag matches package version
id: version
run: |
set -euo pipefail
pkg=$(node -p 'require("./package.json").version')
const=$(sed -n 's/^export const VERSION = "\([^"]*\)";.*/\1/p' src/constants.ts)
if [ -z "$const" ]; then
echo "::error::Could not read VERSION from src/constants.ts."
exit 1
fi
if [ "$pkg" != "$const" ]; then
echo "::error::package.json version ${pkg} does not match src/constants.ts VERSION ${const}."
exit 1
fi
if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
tag=${GITHUB_REF_NAME#v}
if [ "$pkg" != "$tag" ]; then
echo "::error::Tag v${tag} does not match package.json version ${pkg}. Bump package.json and src/constants.ts, then re-tag."
exit 1
fi
fi
echo "version=${pkg}" >> "$GITHUB_OUTPUT"
echo "Versions agree: ${pkg}"

# Checked before publishing so a tag without release notes fails before
# anything reaches npm.
- name: Check CHANGELOG.md has a section for this version
env:
VERSION: ${{ steps.version.outputs.version }}
run: |
set -euo pipefail
if ! grep -qF "## [${VERSION}]" CHANGELOG.md; then
echo "::error::CHANGELOG.md has no '## [${VERSION}]' section."
exit 1
fi

test:
name: Gates (Node ${{ matrix.node-version }})
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 20
strategy:
matrix:
node-version: [18, 20, 22]
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node-version }}
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm typecheck
- run: pnpm lint
- run: pnpm test
- run: pnpm test:types
# `build` runs the `postbuild` check (scripts/assert-no-node-builtins.mjs).
- run: pnpm build

workers:
name: cloudflare worker fixture
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm build
- run: pnpm install --frozen-lockfile
working-directory: fixtures/worker
- run: pnpm exec wrangler deploy --dry-run --outdir .wrangler-build
working-directory: fixtures/worker
- run: pnpm exec vitest run
working-directory: fixtures/worker

publish:
name: Publish to npm
needs: [verify, test, workers]
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
id-token: write # npm trusted publishing (OIDC) and provenance
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: pnpm
registry-url: https://registry.npmjs.org
# Trusted publishing needs npm >= 11.5.1; Node 22 bundles npm 10.
- run: npm install -g npm@^11.5.1
- run: pnpm install --frozen-lockfile
- run: pnpm build

- name: Publish
env:
VERSION: ${{ needs.verify.outputs.version }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
run: |
set -euo pipefail
name=$(node -p 'require("./package.json").name')
# Prereleases (v1.2.0-rc.1) go to the "next" dist-tag, never "latest".
dist_tag=latest
case "$VERSION" in *-*) dist_tag=next ;; esac

# Only a tag push publishes. A manual dispatch (even on a tag ref) is a dry run.
if [ "${GITHUB_EVENT_NAME}" != "push" ] || [ "${GITHUB_REF_TYPE}" != "tag" ]; then
echo "Not a tag push: dry run only."
npm publish --dry-run --access public --tag "$dist_tag"
exit 0
fi

# Lets a failed run be re-run after the npm step already succeeded.
if npm view "${name}@${VERSION}" version >/dev/null 2>&1; then
echo "${name}@${VERSION} is already on npm; skipping publish."
exit 0
fi

# npm tries OIDC trusted publishing first and only falls back to this
# token when no trusted publisher matches. Never echo it.
if [ -n "${NPM_TOKEN}" ]; then
export NODE_AUTH_TOKEN="${NPM_TOKEN}"
echo "NPM_TOKEN secret present (used only if trusted publishing is not configured)."
else
echo "No NPM_TOKEN secret; relying on npm trusted publishing."
fi
npm publish --access public --provenance --tag "$dist_tag"

release:
name: Publish GitHub Release
needs: [verify, publish]
runs-on: ubuntu-latest
timeout-minutes: 5
# Only for real tag pushes; workflow_dispatch runs just the gates + dry run.
if: github.event_name == 'push' && github.ref_type == 'tag'
permissions:
contents: write
steps:
- uses: actions/checkout@v4

- name: Extract release notes from CHANGELOG.md
env:
VERSION: ${{ needs.verify.outputs.version }}
run: |
set -euo pipefail
# Everything after "## [VERSION]" up to the next "## " heading.
awk -v hdr="## [${VERSION}]" '
index($0, hdr) == 1 { found = 1; next }
found && /^## / { exit }
found { print }
' CHANGELOG.md > "${RUNNER_TEMP}/notes.md"
if ! grep -q '[^[:space:]]' "${RUNNER_TEMP}/notes.md"; then
echo "::error::CHANGELOG.md section for ${VERSION} is empty."
exit 1
fi
wc -l "${RUNNER_TEMP}/notes.md"

- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ needs.verify.outputs.version }}
run: |
set -euo pipefail
tag="${GITHUB_REF_NAME}"
repo="${GITHUB_REPOSITORY}"
if gh release view "$tag" --repo "$repo" >/dev/null 2>&1; then
echo "Release $tag already exists; leaving its notes as they are."
exit 0
fi
prerelease=()
case "$VERSION" in *-*) prerelease=(--prerelease) ;; esac
gh release create "$tag" \
--repo "$repo" \
--title "$tag" \
--verify-tag \
--notes-file "${RUNNER_TEMP}/notes.md" \
"${prerelease[@]}"
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ jobs:
cache: pnpm
- run: pnpm install --frozen-lockfile
- run: pnpm typecheck
- run: pnpm lint
- run: pnpm test
- run: pnpm test:types
- run: pnpm build

workers:
Expand Down
Loading
Loading