Skip to content

feat: watch the data proxy through Workers Issues and post a weekly observability digest to Slack - #1

Open
alukach wants to merge 1 commit into
mainfrom
feat/proxy-issues-and-digest
Open

alukach wants to merge 1 commit into
mainfrom
feat/proxy-issues-and-digest

Conversation

@alukach

@alukach alukach commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

sourcebot now watches the data proxy through Cloudflare's Workers Issues, and a new weekly digest reviews the proxy's observability and posts the review to #devops on Slack.

Why

The monitor had never seen a data proxy error. config.yaml named the Worker data-source-coop, but it is source-data-proxy, so every run logged Cloudflare: 0 errors while Vercel errors came through normally. The monitor also stopped altogether on 2026-10-10: GitHub disabled its schedule (disabled_inactivity) after 60 days without repository activity.

Monitor

  • Source: the Cloudflare source reads Workers Observability Issues (/workers/observability/issues), Cloudflare's own grouping of exceptions, 5xx responses and ERROR lines. The old source read up to 500 raw ERROR events per 6-hour window, and the proxy logs about 800 in that time, mostly one known error (data.source.coop#206), so it would have been cut off.
  • Skipped groups: vulnerability-scan detections, and groups first seen before cloudflare_issues_since (2026-11-19). The roughly 20 groups that already exist are tracked by hand (#206, #280, #281), so the first run won't file duplicates.
  • Counts: Issues only carry a running total. Errors from this source are labelled "N times in total since it was first seen", and the D1 total_count is raised to that total rather than added to (MAX(total_count + ?, ?)). Vercel errors keep their per-window counts.
  • Auto-fix: auto_fix is now off for data.source.coop. Most proxy error groups are infrastructure problems (Cloudflare 520/524/525 errors, backend STS failures), not code a bot can fix. A person can still assign Scooter to an issue.

Digest

src/digest.ts runs from .github/workflows/digest.yml on Mondays at 16:00 UTC, or manually with days and a post toggle.

  • Data gathered, through fixed queries rather than by an agent with API access:
    • the data-proxy: monitor definitions and their alert notifications, one per notification (the webhook copy, not the duplicate email);
    • Workers Issues seen in the period;
    • Analytics Engine figures: daily requests, 5xx errors and PUTs, daily read p99, the worst 15-minute read p99 windows, top 5xx by product and status, and the busiest products;
    • Workers Logs WARN/ERROR volume by day, and the top sources with request IDs, durations and object paths collapsed so variants group together;
    • data.source.coop's open issues.
  • Summary: Claude (the anthropic_model setting, one turn, no tools) writes a Slack message of at most about 2,500 characters: which alerts were real, what drove errors and latency, which noise sources an open issue tracks and which are untracked, and up to 5 suggested actions. It only suggests: it files no issues and changes no monitors.
  • Failure handling: a data source that fails, such as a token missing a permission, is reported as unavailable and the digest still posts.
  • Where it posts: the SLACK_WEBHOOK_URL secret. Without it, the digest prints to the workflow log.

Keepalive

Both scheduled workflows run gh workflow enable on themselves at the end of each run (actions: write), so the inactivity rule shouldn't switch them off again. GitHub doesn't document whether re-enabling an enabled workflow resets the 60-day clock, so check after about two months.

Before merging

  • Add the repository secret SLACK_WEBHOOK_URL, a #devops incoming webhook.
  • Add Account → Notifications: Read to the CLOUDFLARE_API_TOKEN token. The digest needs it for alert policies and history; without it, that section reports unavailable.

After merging

  • Re-enable the monitor: gh workflow enable monitor.yml -R source-cooperative/sourcebot.
  • Run Observability Digest manually with post unchecked, and read the output in the log before the first Monday post.

Testing

npx tsc --noEmit passes and npx vitest run passes (40 tests). New tests cover:

  • Workers Issues mapping, filtering and paging;
  • running-total counts in the monitor;
  • the digest's data gathering, alert deduplication, filtering out low-traffic windows, log-message grouping, and continuing when a source fails.

Neither job has been run against the live Cloudflare APIs in this branch. The Workers Issues response fields (title, type, count, firstObserved, lastObserved) and every query were checked against production by hand this week, but the manual digest run above is the first full end-to-end run.

🤖 Generated with Claude Code

…bservability digest to Slack

The monitor had never seen a data proxy error: it queried the Worker as `data-source-coop`, but the Worker is `source-data-proxy`, so every run logged "Cloudflare: 0 errors". It also stopped altogether on 2026-10-10, when GitHub disabled the schedule after 60 days without repository activity.

- The Cloudflare source now reads Workers Observability Issues, Cloudflare's own grouping of exceptions, 5xx responses and ERROR lines, rather than up to 500 raw ERROR events per window, which the proxy's ~800 ERROR lines per 6 hours would overflow. Vulnerability-scan detections are skipped, and so are groups first seen before `cloudflare_issues_since`, which are already tracked by hand. Issues carry a running total, so those errors are reported as "N times in total" and raise `total_count` to that total instead of adding to it.
- `auto_fix` is off for data.source.coop: most proxy error groups are infrastructure (Cloudflare 52x, backend STS), not code a bot can fix.
- New weekly digest (`src/digest.ts`, `digest.yml`, Mondays 16:00 UTC): gathers alert notifications and monitor definitions, Workers Issues, Analytics Engine traffic, error and latency figures, Workers Logs WARN/ERROR sources and the repo's open issues; has Claude write a short review; and posts it to Slack via `SLACK_WEBHOOK_URL`. It only suggests. A data source that fails is reported as unavailable rather than failing the digest.
- Both scheduled workflows re-enable themselves on each run so the inactivity rule cannot switch them off again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant