Projects: a tab for what each team is building

.github/workflows/build.yml

@@ -16,9 +16,21 @@ jobs:
           node-version: "24"
           cache: npm
 
+      # also builds shared/dist and installs frontend-v2 (package.json postinstall)
       - run: npm install
 
       - name: Lint
         run: npm run lint
 
       - run: npm run build
+
+      - name: Server tests
+        run: npm test
+
+      - name: Frontend types
+        working-directory: frontend-v2
+        run: npx tsc --noEmit
+
+      - name: Frontend tests
+        working-directory: frontend-v2
+        run: npx vitest run

README.md

@@ -90,6 +90,189 @@ Two query string parameters are available to filter the displayed pulls:
 
    `ex. https://pulldasher.example.com?milestone=site-redesign,12/5,12/19`
 
+### Projects
+
+An optional tab for whoever plans the work: which projects the open PRs
+serve, who is on each, how they are moving, and a roadmap to plan them on.
+Everything but the roadmap is read from GitHub; something else (a person,
+or a job) files the PRs:
+
+- A PR joins a project through one label, `project:<slug>`. PRs that fit no
+  project get `project:misc`. Keep a label to 32 characters with no spaces:
+  that's what Pulldasher's label table and filter box hold.
+- A project's record is the issue in the projects repo carrying the same
+  label; with no projects repo, names come from labels. The issue's title
+  is the project's name, its
+  assignee the lead, its milestone (and due date) the target,
+  `parent:<slug>` labels its parents (any number), and an `ongoing` label
+  marks work with no end. GitHub's own issue fields show too: its Start
+  date, its Priority, and its Target date, which wins over a milestone,
+  since it's set on that one issue. Close it as completed when it's done, or as not
+  planned when it's dropped or merged into another. Labeling an issue in
+  the projects repo is also how a project starts before it has a PR. An
+  issue with the label in any other repo is work inside the project and
+  never renames it.
+
+Turn it on with the `projects` block in `config.js` (see
+`config.example.js`): a projects repo if you want one, and `developerTeams`, the teams
+whose members count as developers (anyone else with a PR is a
+non-developer, whose work needs a developer's review). The teams can also
+be edited from the People view or the API; saved ones replace
+`developerTeams` until someone goes back to it. Send Issues events in the
+tracked repos' webhook, and give a projects repo the same webhook, so a
+newly labeled issue shows up right away; an hourly pass catches what a
+webhook missed.
+
+The tab has five views and a page per project:
+
+- **Overview**: where the plans stand (the latest update on every item in
+  progress, worst first, which copies as text for a status email),
+  headline numbers (how many calls Decide has waiting, then a date
+  range's numbers, each compared with the same number of days before),
+  whether the backlog is growing (the PRs open each day, and the PRs
+  opened and merged each week), where the time went (the projects that
+  took the most developer-days, a door to Look back), and every
+  project on one list you can sort, group by parent, lead or team,
+  search, and download as CSV, each with how its roadmap plan is going.
+  A project's own page shows its plan and latest update too.
+- **Decide**: the calls owed now, as a list that empties: projects with 3
+  or more PRs (open, or merged in the last 14 days) and no decision, plans
+  past their end or their target date, stalls (open PRs with no activity
+  for 21 days), updates that say at risk or off track, work marked done or
+  dropped (on the roadmap or by closing its issue) whose PRs are still
+  open, and parked work whose PRs moved. Each row says why it's there, with
+  its team, size and target, and takes one click: commit through the end
+  of a coming month or quarter, park, finish, or drop. The row can also say
+  where the work came from, saved with the call. People take turns running
+  the list, a week each: it names this week's and next week's, and the
+  turns can be changed in place or with `PATCH /api/v1/settings`. A team's lead can
+  take just their team's rows, and copy them as text for a meeting's
+  notes. It weighs every project, whatever the filter bar narrows the
+  other views to. The tab's label counts what's left.
+- **Roadmap**: the plan set against everything actually in flight. Across
+  the top, how many projects were in flight each week (from their PRs)
+  and, for the weeks ahead, the plans plus every project big enough to owe
+  a decision that has none, as if nothing changes, against a line at the
+  number of
+  developers. Below, the plans in priority order, then every live project
+  with no plan, one click from being planned.
+  - By month or by quarter; click a quarter or a month to fill the width
+    with it, and zoom out or step to the next one from the toolbar. Or
+    now, next and later, without week dates, for readers outside
+    engineering.
+  - Click a week in the load chart to see only what was in flight then, or
+    a count to see only those ("with no plan"). Find narrows by name,
+    lead or team.
+  - Each plan can say where its work came from: asked for from above
+    (top-down), or found by the team, as a fire to put out or its own pick
+    (bottom-up). Once the rows are narrowed to the plans, the load chart
+    splits its roadmap count that way, and a click on one shows only those
+    plans.
+  - Drag rows to set priority, drag a bar to move it and its edge to change
+    its length. A click on a bar opens the plan's details under its row,
+    where Decide's calls commit it through a month or a quarter in one
+    click, and Edit details holds the rest.
+    Team lanes say the most each team has in flight at once against its
+    developers, and draw a dashed line where, in priority order, the work
+    in flight outnumbers them: what's below it is what to park.
+  - A linked plan still in flight past its end grows an amber "+3 wk over";
+    its project's milestone is a flag with its date. An item can wait on
+    others, and says so when its plan starts before one of them ends.
+    Parked work is stopped for now without being dropped, and leaves the
+    load.
+  - Each item takes updates: on track, at risk or off track, and a note,
+    kept as a history. Work in progress with no update for 14 days is
+    flagged. The roadmap is the one thing the tab stores itself, in the
+    `roadmap_items` and `roadmap_updates` tables.
+- **People**: developers by team and everyone else: live projects each,
+  open PRs, PRs opened and merged, and reviews given, including how many
+  went to non-developers' PRs.
+- **Look back**: where the time went over a date range, for a retro, in
+  developer-days rather than PR counts, so a month of work outweighs a
+  one-line fix. A developer-day is a day someone opened, merged, commented
+  on, stamped or reviewed a PR, split across the PRs they touched that
+  day, as writing (their own) or reviewing (anyone else's). Commits aren't
+  counted: in a sample of 60 merged PRs they added about 4% more days,
+  since PRs open and merge within a day or two.
+  - Tiles open it, each against the same number of days before: the
+    days, how much went to reviewing, to work on the roadmap, to fires,
+    to PRs filed to no project, and how many different projects each
+    person touched a week.
+  - One list splits the days by project, by where the work came from, by
+    whose PR it was (their own, other developers', non-developers',
+    bots'), by team, by person or by repo, and can keep only the writing
+    or the reviewing days. Each row draws its days week by week on one
+    scale, with its share then and now, and how many of its PRs merged.
+  - A row's name goes where it names: a project to its page, an origin to
+    its plans, a team or person to only their days. Anywhere else on the
+    row opens the PRs that took the most days and who spent them. It all
+    copies as text for the retro's notes.
+
+To put the plans where people already look, `bin/sync-issue-fields` copies
+each plan under way into its project issue's Start date, Target date and
+Priority fields: the plan's first Monday and last day, and Urgent for a
+fire, else High, Medium or Low for work now, next or later. A team's project
+board shows those fields, and the issue's history keeps every change. It
+fills a field nobody has set, and changes one only if its own token set it
+last, so a date a person moved stays moved; it says so instead. It's a dry
+run unless given `--apply`, and its writes show as `github.token`'s account.
+
+```sh
+bin/sync-issue-fields           # what it would write
+bin/sync-issue-fields --apply   # write it
+```
+
+Everything the tab shows, and every change the roadmap takes, is in the API,
+Bearer-authed with the caller's own GitHub token like `/api/v1/pulls`, so a
+script or a Claude session can read the projects and run the roadmap
+without a browser. `GET /api/v1` lists every route with what it does, and
+the rules a roadmap write is checked against:
+
+```sh
+curl -s -H "Authorization: Bearer $(gh auth token)" https://pulldasher.example.com/api/v1
+```
+
+- `GET /api/v1/projects` and `GET /api/v1/people`: each project and each
+  person with where they stand today and their numbers for a window. Both
+  take `start` and `end` as `YYYY-MM-DD` days (default: the last 30 days, at
+  most 400), and `project=<slug>` narrows the numbers to one project.
+- `GET /api/v1/decide`: the calls owed now, worst first, each with its
+  project, its roadmap item if it has one, and its reasons. `GET /api/v1`
+  says what each reason means and which write clears it. It also names who
+  runs Decide this week and next.
+- `GET /api/v1/retro`: where the days went in a window (`start`, `end`,
+  default the last 30 days): per person and PR, their share of active
+  days, whether they wrote it, and its project.
+- `GET /api/v1/updates-owed`: each lead who owes an update, with their
+  plans in progress that have gone 14 days without one, longest overdue
+  first. It's what a weekly reminder would send each lead; nothing sends
+  it yet.
+- `GET /api/v1/load`: projects in flight each week against the developer
+  count, on the roadmap and not, with the weeks after this one counted as
+  if nothing changes. `start` and `end` as above (default: 12 weeks back to
+  26 ahead).
+- `GET /api/v1/roadmap` and `GET /api/v1/roadmap/:id`: the items in
+  priority order, or one, each with its latest update.
+- `POST /api/v1/roadmap`, `PATCH` and `DELETE /api/v1/roadmap/:id`: add,
+  change or remove an item. A removed item is kept, out of every list, and
+  `PATCH` with `{"restore": true}` puts it back with its updates, place and
+  waits. `POST /api/v1/roadmap/:id/move` with
+  `{"before": <id>}` (or `null` for the bottom) moves one item without
+  sending the whole order; `PUT /api/v1/roadmap/order` sets the whole order.
+- `GET` and `POST /api/v1/roadmap/:id/updates`: an item's updates, or a new
+  one, `{"health": "at_risk", "body": "..."}`. `DELETE
+  /api/v1/roadmap/:id/updates/:update` takes one back; only its author can.
+- `GET` and `PATCH /api/v1/settings`: the developer teams and whether
+  they're saved or `config.js`'s. `{"developer_teams": {"Store": ["dana"]}}`
+  replaces them; `null` goes back to `config.js`.
+
+A write takes a JSON body and is recorded as the token's login. The board
+itself uses the same handlers through `/roadmap`, with its session.
+
+Pulldasher also re-lists every tracked repo's open PRs once an hour and
+refreshes only the ones its database has wrong, so a lost webhook can't leave
+a merged PR counted as open.
+
 ## Architecture
 
 When first started, the Pulldasher server fetches information about the current

app.js

@@ -15,7 +15,15 @@ import mainController from './controllers/main.js';
 import hooksController from './controllers/githubHooks.js';
 import statsController from './controllers/stats.js';
 import userNamesController from './controllers/user-names.js';
-import apiController from './controllers/api.js';
+import projectsController from './controllers/projects.js';
+import roadmapController, { canWrite, forgetProjectCaches } from './controllers/roadmap.js';
+import { API_ROUTES, apiIndex } from './controllers/api-routes.js';
+import settingsController from './controllers/settings.js';
+import { syncRepoIssues } from './lib/project-issue-sync.js';
+import { loadSettingsWithRetry } from './lib/settings.js';
+import { issueRepos, projectSettings } from './lib/projects.js';
+import { setOnWorkSynced, syncWork } from './lib/work.js';
+import { missingMigrations } from './lib/schema-check.js';
 import apiAuth from './lib/api-auth.js';
 import Debug from './lib/debug.js';
 import { createServer } from 'http';
@@ -85,21 +93,81 @@ app.use('/', express.static(__dirname + '/frontend-v2/dist'));
 app.get('/token', mainController.getToken);
 app.get('/stats-history', statsController.getHistory);
 app.get('/user-names', userNamesController.getNames);
+app.get('/projects-data', projectsController.getBoardData);
+app.get('/retro-data', projectsController.getRetro);
+app.get('/work-data', projectsController.getWork);
+app.get('/issue-search', projectsController.searchIssues);
+app.get('/project-work', projectsController.getProjectWork);
+// the review board's view of projects: which projects its PRs link, and
+// each project's plan, so it orders and names PRs by the Projects tab's rule
+app.get('/project-standing', roadmapController.standing);
+
+// Something open boards show of the projects changed: what's kept of them is
+// read again, and every open board fetches again, so one person's call shows
+// on another's screen without a reload.
+function projectsChanged() {
+   forgetProjectCaches();
+   pullManager.projectsChanged();
+}
+// every write the Projects tab can see
+const PROJECT_WRITES = ['/project-issues', '/roadmap', '/settings', '/api/v1/project-issues', '/api/v1/roadmap', '/api/v1/settings'];
+app.use(PROJECT_WRITES, function (req, res, next) {
+   if (req.method !== 'GET') {
+      res.on('finish', function () {
+         if (res.statusCode < 300) projectsChanged();
+      });
+   }
+   next();
+});
+setOnWorkSynced(projectsChanged);
+app.post('/project-issues', canWrite, projectsController.attachIssue);
+app.delete('/project-issues', canWrite, projectsController.detachIssue);
+// the roadmap is the one part of the Projects tab people edit here: reads are
+// gated like the other board data (lib/authentication.js), writes by canWrite
+app.get('/roadmap', roadmapController.list);
+app.post('/roadmap', canWrite, roadmapController.create);
+app.put('/roadmap/order', canWrite, roadmapController.reorder);
+app.patch('/roadmap/:id', canWrite, roadmapController.update);
+app.delete('/roadmap/:id', canWrite, roadmapController.remove);
+app.get('/roadmap/:id/updates', roadmapController.updates);
+app.post('/roadmap/:id/updates', canWrite, roadmapController.postUpdate);
+app.delete('/roadmap/:id/updates/:update', canWrite, roadmapController.removeUpdate);
+app.get('/settings', settingsController.get);
+app.patch('/settings', canWrite, settingsController.update);
 app.post('/hooks/main', hooksController.main);
 
-// /api/v1: machine-to-machine JSON for the review skills, Bearer-authed with
-// the caller's own GitHub token (see lib/api-auth). Independent of the
-// cookie-session gate -- setupRoutes never registers these paths, so the
-// session `auth` middleware doesn't run for them.
-app.get('/api/v1/me', apiAuth, apiController.getMe);
-app.get('/api/v1/pulls', apiAuth, apiController.getPulls);
+// /api/v1: machine-to-machine JSON for the review skills and scripts,
+// Bearer-authed with the caller's own GitHub token (see lib/api-auth). Every
+// route is in controllers/api-routes.js, and GET /api/v1 lists them.
+// Independent of the cookie-session gate -- setupRoutes never registers these
+// paths, so the session `auth` middleware doesn't run for them.
+app.get('/api/v1', apiAuth, apiIndex);
+for (const { method, path, handlers } of API_ROUTES) app[method](path, apiAuth, ...handlers);
 
 // Warm the bot-login cache (used to tell a pulldasher claim apart from a
 // GitHub-UI self-request) before any webhook or socket traffic needs it.
 // Memoized in git-manager, so this just avoids the first caller paying for
 // the lookup.
 git.getBotLogin();
 
+// Saved settings (the developer teams) replace config.js's once loaded;
+// until then, and if the table can't be read, config.js's stand.
+loadSettingsWithRetry();
+
+// Migrations run by hand, before a deploy (migrations/*.sql never run on
+// one). Say loudly at startup which are missing, since the board then
+// fails in pieces: pull refreshes, the issue syncs, roadmap reads.
+missingMigrations()
+   .then(missing => {
+      if (missing.length) {
+         console.error(
+            'The database is missing migrations, so parts of the board will fail. Run these by hand, in order: %s',
+            missing.join(', ')
+         );
+      }
+   })
+   .catch(err => console.error('Checking the schema failed: %s', (err && err.message) || err));
+
 debug('Loading all recent pulls from the DB');
 dbManager
    .getRecentPulls(pullManager.getOldestAllowedPullTimestamp())
@@ -114,20 +182,61 @@ dbManager
    .then(function () {
       debug('Refreshing all open pulls from the API');
       refresh.openPulls().finally(boardLoaded);
+      syncProjectIssues();
+      syncAttachedIssues();
    })
    .done();
 
 // Webhooks get lost, and a lost `closed` left a PR open on the board until the
 // next restart (pulldasher#501 repairs it only at startup), which inflates every
 // backlog number. Once an hour, list each repo's open pulls and refresh just the
-// ones the DB has wrong.
+// ones the DB has wrong, and pick up project issues that changed.
 const RECONCILE_MS = 60 * 60 * 1000;
 setInterval(function () {
    refresh.reconcileOpenPulls().catch(function (err) {
       console.error('Hourly open-pull repair failed: %s', (err && err.message) || err);
    });
+   syncProjectIssues();
+   syncAttachedIssues();
 }, RECONCILE_MS);
 
+// The issues added to projects by hand, and which PRs link each issue a
+// project has (lib/work.js), read off GitHub once an hour; a webhook on one
+// of those issues reads them sooner.
+function syncAttachedIssues() {
+   syncWork(projectSettings()).catch(function (err) {
+      console.error('Work sync failed: %s', (err && err.message) || err);
+   });
+}
+
+// Project issues can live in any tracked repo (lib/projects.js issueRepos).
+// The projects repo is synced whole at startup; a tracked repo only from
+// TRACKED_ISSUES_DAYS back, since listing every issue of a big repo costs
+// thousands of calls, and webhooks already keep its issues current. So an
+// issue labeled while webhooks were down for longer than that waits for its
+// next change (or bin/refresh-open-issues). After that, only what changed.
+// `since` backs off a few minutes so GitHub's clock can't skip an update.
+const TRACKED_ISSUES_DAYS = 7;
+function syncProjectIssues() {
+   const projects = projectSettings();
+   if (!projects) return;
+   const startedAt = new Date(Date.now() - 5 * 60 * 1000).toISOString();
+   const lookback = new Date(Date.now() - TRACKED_ISSUES_DAYS * 86400 * 1000).toISOString();
+   const synced = issueRepos(projects).map(function (repo) {
+      const firstSince = repo === projects.repo ? null : lookback;
+      return syncRepoIssues(refresh, repo, firstSince, startedAt).catch(function (err) {
+         console.error('Project issue sync failed in %s: %s', repo, (err && err.message) || err);
+         return false;
+      });
+   });
+   // a project's name, target or state may have changed: open boards hear it
+   // once, after every repo is read, and only when an issue changed (a word
+   // per repo had every open Projects tab fetch it all again once per repo)
+   Promise.all(synced).then(function (changed) {
+      if (changed.some(Boolean)) projectsChanged();
+   });
+}
+
 //====================================================
 // Socket.IO
 const io = new Server(httpServer);