feat: observe enrolled GitHub project state and activity

cloudflare/GITHUB.md

@@ -203,6 +203,66 @@ See the official [GitHub license API](https://docs.github.com/en/rest/licenses/l
 [GraphQL blobs](https://docs.github.com/en/graphql/reference/git#blob), and
 [App metadata API](https://docs.github.com/en/rest/apps/apps#get-the-authenticated-app).
 
-The project-state quest owns ongoing snapshots, branches, PRs, and event
-reconciliation; this setup preview does not reserve or launch ready quests.
+The setup preview does not reserve or launch ready quests.
 The jobs/funding quests own live eligibility checks and contribution execution.
+
+## Project state and activity
+
+An enrolled project's setup page links to `/projects/github/<owner>/<repo>/state`.
+A maintainer explicitly refreshes observations with its same-origin form. The
+service rereads the default branch at an exact commit using the preview's shared
+snapshot reader and validates the Quest tree with the pure core. The authenticated
+board and nested quest pages use the existing `Project` seam. Source links retain
+the exact commit; live project state never replaces GitHub as authority.
+
+Each refresh reconciles one page of native issues, **all** PRs (including human
+contributions and closed/merged PRs), and `quest/` branches. Independent opaque
+cursors persist across requests. Continue reconciliation until all connections
+finish; candidates remain empty while a scan is incomplete or a webhook has
+invalidated it. The next completed scan also detects deleted entities and removed
+claims. Refreshing a completed scan starts again from the authoritative entity
+connections, so missed webhooks and later changes to old entities are recovered.
+This deliberately avoids GitHub's bounded, delayed activity feed; GitHub does
+[not automatically redeliver failed webhooks](https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries).
+There is no scheduled background poll in this stage.
+
+A single GraphQL activity query reads up to 50 issues, 50 PRs and 100 branch
+claims, keeping external request count independent of entity/file counts. Bulk
+D1 writes use `json_each`, avoiding a query per entity or SQLite's parameter
+limit. A revision comparison commits each page and cursor atomically; concurrent
+refreshes or an intervening webhook cannot publish over a newer observation.
+A webhook restarts an unfinished scan on the next refresh. Installation tokens
+remain read-only, repository-scoped, and revoked after use.
+
+Durable activity items use the GitHub repository ID, entity kind and full database
+ID. GitHub's GraphQL `fullDatabaseId` strings and webhook numeric IDs map to the
+same stored text key. Duplicate delivery IDs, redelivery with new IDs, later
+pushes/comments/reviews, and reconciliation all update the same item. Webhooks
+retain identity pointers only, never issue/PR bodies. Authoritative API reads add
+escaped titles, state, comment/review counts, current PR head, that head's check
+rollup, aggregate review decision, and up to 20 latest reviewer records with an
+explicit pagination flag. Raw bodies and individual check logs are not cached.
+Unavailable/deleted entities retain their notification identity and are marked
+missing rather than disappearing or presenting old state as current.
+
+A PR associates with a quest only when its head branch exactly matches the quest's
+path without `.md` (or an epic's `/README.md`). An unrelated human PR remains in
+activity without a guessed association. Open associated PRs and observed quest
+branch claims exclude that quest from candidate data; branch claims display as
+Claimed on the board. Runner/run associations belong to job coordination.
+
+Every cached read and refresh proves current maintainer authority, live repository
+identity/public visibility, installation access, and enrollment for that
+installation. Refresh rechecks those boundaries after provider reads. Paused
+projects remain readable but have no candidates. Disconnect cascades cached
+snapshots and activity; explicitly enrolling a replacement installation clears
+prior observations. No positive authority cache is created. Candidate data is an
+observation tied to its commit, **never execution permission**: job coordination
+must validate current eligibility, fresh revision, locks, and PRs before a start.
+Review/check data is similarly descriptive and cannot approve a merge.
+
+The `/data` endpoint exposes the same authenticated snapshot/activity with
+`Cache-Control: no-store` for future decision and job consumers. The activity
+page is a secondary read view; the durable decision inbox and its unread/answer
+state belong to the prompts workflow. Neither incoming activity nor refreshing
+starts a model, creates a quest/job, spends usage, or writes to GitHub.

cloudflare/README.md

@@ -26,8 +26,8 @@ renders the repository's `docs/*.md` into `build/docs.json`, so the Markdown
 stays the single source and its relative links keep working on GitHub: links to
 other docs become `/docs/<name>` and other repository paths point at GitHub. It
 also snapshots this repository's `quest/` tree and finished quests into
-`build/board.json`, which is the board's only project until the GitHub project
-state integration supplies enrolled repositories. A shallow clone has no
+`build/board.json`, which supplies the public board. Enrolled repositories use authenticated
+GitHub snapshots through the same board model. A shallow clone has no
 history, so its board shows no finished work. It also copies `design/theme/` and `src/site.css` into `build/public/`, which the
 Worker serves as static assets (`/theme/theme.css`, `/theme/logo.svg`,
 `/theme/og.png`, and `/site.css`). `build/` is ignored by Git.
@@ -65,7 +65,7 @@ The passkey relying party is the hostname of `AUTH_URL`, and its exact origin
 is the only one accepted. Changing the hostname orphans existing passkeys;
 people enroll new ones after signing in with a provider.
 
-GitHub remains the planned authority for enrolled project state. The Worker
+GitHub is authoritative for enrolled project state. The Worker
 has no Durable Objects; D1 holds accounts, sessions, and GitHub App state.
 
 `cloudflare/` is a Bun workspace of the repository root, so the Worker

cloudflare/migrations/0006_project_state.sql

@@ -0,0 +1,24 @@
+-- Cached observations never grant authority. Disconnect removes their retention.
+CREATE TABLE github_project_state (
+  repository INTEGER PRIMARY KEY REFERENCES github_projects(id) ON DELETE CASCADE,
+  revision INTEGER NOT NULL DEFAULT 0,
+  epoch TEXT NOT NULL,
+  invalidated INTEGER NOT NULL DEFAULT 0 CHECK(invalidated IN (0,1)),
+  snapshot TEXT,
+  scan TEXT,
+  observedAt INTEGER
+);
+-- One durable notification per provider entity, independent of delivery IDs.
+-- Webhooks keep only identity pointers; authenticated reconciliation fills data.
+CREATE TABLE github_project_activity (
+  repository INTEGER NOT NULL REFERENCES github_projects(id) ON DELETE CASCADE,
+  kind TEXT NOT NULL CHECK(kind IN ('issue','pull')),
+  id TEXT NOT NULL,
+  number INTEGER NOT NULL,
+  firstSeen INTEGER NOT NULL,
+  changedAt INTEGER NOT NULL,
+  data TEXT,
+  seenIn TEXT,
+  missing INTEGER NOT NULL DEFAULT 0,
+  PRIMARY KEY(repository,kind,id)
+);

cloudflare/src/board/model.ts

@@ -10,7 +10,12 @@ import type { Change, Completed, Project } from "./project";
  * A quest's most advanced state, per design/theme.md: Ready to turn in, then
  * Available or Requires.
  */
-export type Status = { kind: "turn-in"; change: Change } | { kind: "available" } | { kind: "blocked"; requires: Ref[] };
+export type Status =
+	| { kind: "turn-in"; change: Change }
+	| { kind: "available" }
+	| { kind: "pending" }
+	| { kind: "accepted" }
+	| { kind: "blocked"; requires: Ref[] };
 
 /** A link to another quest, or a blocker's own words when it names none. */
 export type Ref = { title: string; href: string | null };
@@ -64,10 +69,14 @@ export function title(text: string): { size: Size | null; name: string } {
  * A quest's page: its path without `.md`, like its branch name. A README is
  * its directory's page.
  */
+export function boardHref(project: Project): string {
+	return project.href ?? `/repos/${project.name}`;
+}
+
 export function questHref(project: Project, path: string): string {
-	if (path === ROOT) return `/repos/${project.name}`;
+	if (path === ROOT) return boardHref(project);
 	const page = path.replace(/\/README\.md$/, "").replace(/\.md$/, "");
-	return `/repos/${project.name}/${encodePath(page)}`;
+	return `${boardHref(project)}/${encodePath(page)}`;
 }
 
 /**
@@ -114,6 +123,9 @@ export function readBoard(project: Project): Board {
 	const docs = new Map(parsed.map((doc) => [doc.path, doc]));
 	const contents = new Map(sorted.map(({ path, content }) => [path, content]));
 	const open = new Set(ready(parsed));
+	// Observations narrow the tree's ready quests; until they are complete, an
+	// unseen change or claim may already occupy one.
+	const free = project.available === undefined ? open : new Set(project.available ?? []);
 	const changes = new Map(project.changes.map((change) => [change.quest, change]));
 
 	const name = (path: string) => {
@@ -124,7 +136,9 @@ export function readBoard(project: Project): Board {
 	const status = (doc: Doc): Status => {
 		const change = changes.get(doc.path);
 		if (change) return { kind: "turn-in", change };
-		if (open.has(doc.path)) return { kind: "available" };
+		if (project.claims?.includes(doc.path)) return { kind: "accepted" };
+		if (free.has(doc.path)) return { kind: "available" };
+		if (open.has(doc.path)) return { kind: "pending" };
 		const requires = blockers(parsed, doc).map((blocker) =>
 			blocker.path === null
 				? { title: blocker.text, href: null }
@@ -214,7 +228,7 @@ export function readBoard(project: Project): Board {
 	}
 
 	const all = milestones.flatMap((milestone) => flatten(milestone.items));
-	const counts = { "turn-in": 0, available: 0, blocked: 0 };
+	const counts = { "turn-in": 0, available: 0, pending: 0, blocked: 0, accepted: 0 };
 	for (const q of all) counts[q.status.kind]++;
 
 	const milestoneOf = (path: string) => milestones.find((c) => c.number !== "" && path.startsWith(`${dir(c.path)}/`));

cloudflare/src/board/pages.tsx

@@ -5,6 +5,7 @@ import { type Crumb, Difficulty, Marker, Paste, SignIn, Token, type User } from
 import { render } from "./markdown";
 import {
 	type Board,
+	boardHref,
 	type Milestone,
 	milestoneAnchor,
 	blobHref,
@@ -27,8 +28,12 @@ function label(status: Status, limit = Infinity): string {
 	switch (status.kind) {
 		case "turn-in":
 			return "Ready to turn in";
+		case "accepted":
+			return "Claimed";
 		case "available":
 			return "Available";
+		case "pending":
+			return "Availability pending";
 		case "blocked": {
 			const titles = status.requires.map((ref) => ref.title);
 			const shown = titles.slice(0, limit).join(", ");
@@ -156,13 +161,15 @@ function FinishedList(props: { finished: Finished[] }) {
 /** The statuses a visitor can filter the board by, in the order a contributor asks about them. */
 export const SHOWN: [Status["kind"], string][] = [
 	["available", "Available"],
+	["pending", "Availability pending"],
+	["accepted", "Claimed"],
 	["turn-in", "Ready to turn in"],
 	["blocked", "Requires others"],
 ];
 
 // Counts by status, which double as filters and as the legend for the markers.
 function Tally(props: { board: Board; show: Status["kind"] | null }) {
-	const base = `/repos/${props.board.project.name}`;
+	const base = boardHref(props.board.project);
 	return (
 		<nav aria-label="Filter quests by status">
 			<ul class="tally">
@@ -171,7 +178,7 @@ function Tally(props: { board: Board; show: Status["kind"] | null }) {
 						All
 					</a>
 				</li>
-				{SHOWN.map(([kind, name]) => (
+				{SHOWN.filter(([kind]) => kind !== "pending" || props.board.counts.pending > 0).map(([kind, name]) => (
 					<li class={`ql-${kind}`}>
 						<a href={`${base}?show=${kind}`} aria-current={props.show === kind ? "page" : undefined}>
 							<Marker status={kind} />
@@ -208,17 +215,26 @@ export function BoardPage(props: { board: Board; show: Status["kind"] | null })
 					)}
 					.
 				</p>
+				{project.available === null && (
+					<p class="muted">
+						Reconciliation with GitHub is incomplete, so availability is pending until every open change and
+						quest branch is observed.
+					</p>
+				)}
 			</header>
 
 			{milestones.map(({ milestone, items }) => (
 				<MilestoneSection milestone={milestone} items={items} />
 			))}
-			{milestones.length === 0 && (
-				<p class="ql-panel muted">
-					No quests are {SHOWN.find(([kind]) => kind === show)![1].toLowerCase()} right now.{" "}
-					<a href={`/repos/${project.name}`}>Show every quest</a>.
-				</p>
-			)}
+			{milestones.length === 0 &&
+				(show ? (
+					<p class="ql-panel muted">
+						No quests are {SHOWN.find(([kind]) => kind === show)![1].toLowerCase()} right now.{" "}
+						<a href={boardHref(project)}>Show every quest</a>.
+					</p>
+				) : (
+					<p class="ql-panel muted">This project has no quests yet.</p>
+				))}
 
 			<div class="board-foot">
 				<section class="ql-panel" aria-labelledby="finished-title">
@@ -256,7 +272,7 @@ const HEADINGS: Record<string, string> = { Goal: "Objectives", Closes: "Rewards"
 export function questCrumbs(board: Board, doc: Doc): Crumb[] {
 	const milestone = (group: Group) => board.milestones.find((c) => c.path === group.path && c.number !== "");
 	return [
-		{ label: board.project.title, href: `/repos/${board.project.name}` },
+		{ label: board.project.title, href: boardHref(board.project) },
 		...trail(board, doc.path).map((group) => ({
 			label: milestone(group) ? `Milestone ${milestone(group)!.number}` : group.title,
 			href: group.href,
@@ -321,12 +337,30 @@ function Actions(props: { board: Board; quest: Quest; user: User | null; path: s
 			</Panel>
 		);
 	}
+	if (status.kind === "accepted")
+		return (
+			<Panel title="Claimed">
+				<p>
+					A quest branch already claims this work. Review its current GitHub state before starting another
+					implementation.
+				</p>
+			</Panel>
+		);
+	if (status.kind === "pending")
+		return (
+			<Panel title="Availability pending">
+				<p class="muted">
+					Reconciliation with GitHub is incomplete, so an open change or quest branch may already claim this
+					quest. Check again once every page is observed.
+				</p>
+			</Panel>
+		);
 	if (status.kind === "blocked") {
 		return (
 			<Panel title="Not yet available">
 				<p class="muted">
 					This quest opens once {status.requires.length === 1 ? "the quest" : "everything"} it requires is
-					complete. Until then, pick an <a href={`/repos/${board.project.name}?show=available`}>available</a>{" "}
+					complete. Until then, pick an <a href={`${boardHref(board.project)}?show=available`}>available</a>{" "}
 					quest.
 				</p>
 			</Panel>
@@ -435,7 +469,7 @@ export function QuestPage(props: { board: Board; doc: Doc; content: string; user
 				)}
 				<section class="panel-links">
 					<p>
-						<a href={`/repos/${project.name}`}>Back to the quest log</a>
+						<a href={boardHref(project)}>Back to the quest log</a>
 						{markdown && (
 							<>
 								{" · "}

cloudflare/src/board/project.ts

@@ -36,11 +36,20 @@ export type Issue = { path: string; title: string };
 export type Project = Snapshot & {
 	/** The project's name in URLs: `/repos/<name>`. */
 	name: string;
+	/** An authenticated board may live under project setup. */
+	href?: string;
 	/** How the project names itself, such as `kixelated/quest`. */
 	title: string;
 	/** Where the repository's other files can be read, or `null` if nowhere public. */
 	web: string | null;
 	changes: Change[];
+	/** Observed branch claims reserve work without implying a finished PR. */
+	claims?: string[];
+	/**
+	 * Quests observations show free to start, or `null` while reconciliation is
+	 * incomplete. Absent when the quest tree alone decides.
+	 */
+	available?: string[] | null;
 	issues: Issue[];
 };