Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/544-schedule-delivery.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The schedule console now reports what actually happened to a run's notifications instead of the fixed `unknown`. A notification a job queued carries that job's id, so the console reads the outbox for the newest decided instant and names the state: sent, queued, sending, retry scheduled, failed or blocked. A finished run that queued no external notification reads `none`, a send that may have crossed the boundary reads `unconfirmed` and is never resent automatically, and a run whose job is still running reads `unknown` — a job queues its notifications only when it ends, so a missing value is never printed as a definite none.
26 changes: 21 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -624,8 +624,20 @@ recorded". `scripts/restore-drill.mjs --status-file` is the writer.
schedule, the next run from `nextTime`, the newest decided instant with its
state, reason, `missed` and `attempts`, whether a live lease holds it, and the
last instant whose job succeeded. A value nobody recorded prints `unknown`: a
schedule with no occurrence row has not run, and this slice has no delivery
outbox, so neither is a zero. The eight facts render through `.schedule-facts`,
schedule with no occurrence row has not run, so it is not a zero. The delivery
fact is read from the outbox (`notification_outbox.job_id`, below) for the job
the newest decided instant dispatched: a job that queued a send reports that
send's state, and a finished job that queued none prints `none` — a run with
nothing to deliver — rather than the `unknown` of a page that did not look. A
job queues its notifications only when it ends (`succeeded`/`failed`), so a
run whose job is still queued or running prints `unknown` too: the empty read
is a value nobody knows yet, and a missing value is never printed as a
definite none. Only a decided instant answers: `skipped` queued nothing by
definition, while `claimed` (a worker holds it, `job_id` still NULL until the
job starts) and `expired` (a lease that lapsed before anyone settled it) have
not decided yet and print `unknown`. The outbox's own
`unknown` (a send that may have crossed the external boundary) is printed as
`unconfirmed`, so it is never confused with "nobody read the outbox". The eight facts render through `.schedule-facts`,
a two-column grid with muted labels that stacks to one column below 640px, so
the console does not fall back to the browser's default definition-list text.
`src/schedules/console.ts` is that view model. The
Expand Down Expand Up @@ -658,9 +670,13 @@ recorded". `scripts/restore-drill.mjs --status-file` is the writer.
(`#387`) must move the count inside that claim, or two workers could each spend
the last slot.
- **The notification outbox** (#544, slice 7): the notifications it covers are
durable rows (`notification_outbox`) instead of a call whose failure lived only
in a log line — the digests, question mails, ops alerts and per-job/PR webhooks
are queued with an idempotency key, so a replayed callback or a second tick
durable rows (`notification_outbox`), and a send that reports on a job carries
its `job_id` (nullable, indexed) so the schedule console can name a run's
delivery without parsing the idempotency key; a row written before the column
existed keeps NULL and is never retro-attached to a run. A notification is a
row instead of a call whose failure lived only in a log line — the digests,
question mails, ops alerts and per-job/PR webhooks are queued with an
idempotency key, so a replayed callback or a second tick
keeps one row rather than sending twice. Per-job success/failure mail and the
transactional approval, quota, trial and feedback mail still send inline in
this slice; they are not queued rows yet. A tick pass sweeps lapsed `sending`
Expand Down
4 changes: 4 additions & 0 deletions src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1773,6 +1773,10 @@ export async function buildApp(options: BuildOptions): Promise<BuiltApp> {
occurrences: store.listScheduleOccurrences(repo.id, 20),
now,
jobStatus: (jobId) => store.getJob(jobId)?.status ?? null,
// #544: the delivery fact is the state of the notification the newest run's
// job queued, read from the outbox rather than assumed unknown. A job that
// queued nothing answers empty, which the console prints as `none`.
jobDeliveries: (jobId) => store.notificationsForJob(jobId),
// #544 slice 6: the same count function the worker charges with, read over
// today's UTC day — the worker charges the *instant's* day, so a catch-up
// run can spend a past day while this fact answers "how much of today is
Expand Down
38 changes: 35 additions & 3 deletions src/db.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ const { DatabaseSync } = require('node:sqlite') as { DatabaseSync: new (path: st
// restore check (#652) has to prove a fixture written by an older build still
// migrates to exactly this number, and a literal in two places would let the
// check and the migration disagree.
export const SCHEMA_VERSION = 47
export const SCHEMA_VERSION = 48

export type UserStatus = 'onboarding' | 'pending' | 'approved' | 'rejected'
export type UserRole = 'user' | 'admin' | 'superadmin'
Expand Down Expand Up @@ -343,6 +343,10 @@ export interface NotificationOutboxRow {
/** The account the email belongs to; null for a webhook. Delivery re-checks
* that account and its current notification address before sending. */
user_id: string | null
/** #544: the job whose outcome this notification reports, when it reports one.
* A scheduled run's delivery is read through it, so the console never has to
* parse the idempotency key to guess which send belongs to which run. */
job_id: string | null
payload_json: string
/** Reused by a human resend: the same logical notification, not a second one. */
idempotency_key: string
Expand Down Expand Up @@ -2189,6 +2193,9 @@ CREATE TABLE IF NOT EXISTS notification_outbox (
transport TEXT NOT NULL,
destination TEXT NOT NULL,
user_id TEXT REFERENCES users(id) ON DELETE CASCADE,
-- #544: the job this send reports, when it reports one. Null for a send that
-- is not about a job (a digest, an ops alert without a job).
job_id TEXT,
payload_json TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
state TEXT NOT NULL,
Expand Down Expand Up @@ -2521,6 +2528,7 @@ export class Store {
this.migrateScheduleMisfireColumns()
this.migrateScheduleBudgetColumns()
this.migrateSchedulePauseColumns()
this.migrateNotificationJobColumns()
this.migrateReleaseColumns()
this.migrateIndexes()
this.backfillTokenRepoIds()
Expand Down Expand Up @@ -2566,6 +2574,17 @@ export class Store {
if (!columns.includes('paused_at')) this.db.exec('ALTER TABLE schedules ADD COLUMN paused_at INTEGER')
}

// #544: the console answers a scheduled run's delivery through the job the
// occurrence dispatched, so an outbox row has to name that job. The column is
// added in place with no backfill: a row written before it cannot be attached
// to a run afterwards, and guessing from the idempotency key would attach the
// wrong one. Such rows answer `none` for an old run, never a fabricated state.
private migrateNotificationJobColumns(): void {
const columns = this.all<{ name: string }>('PRAGMA table_info(notification_outbox)').map((column) => column.name)
if (!columns.includes('job_id')) this.db.exec('ALTER TABLE notification_outbox ADD COLUMN job_id TEXT')
this.run('CREATE INDEX IF NOT EXISTS notification_outbox_job_idx ON notification_outbox(job_id, created_at DESC)')
}

private migrateScheduleOccurrenceColumns(): void {
const columns = new Set(this.all<{ name: string }>('PRAGMA table_info(schedule_occurrences)').map((row) => row.name))
const additions: Array<[string, string]> = [
Expand Down Expand Up @@ -5881,21 +5900,24 @@ export class Store {
transport: NotificationTransport
destination: string
userId?: string | null
/** #544: the job this send reports, so a scheduled run's delivery is readable. */
jobId?: string | null
payload: unknown
idempotencyKey: string
now?: number
}): { row: NotificationOutboxRow; created: boolean } {
const now = input.now ?? Date.now()
const result = this.db.prepare(
`INSERT INTO notification_outbox
(id, transport, destination, user_id, payload_json, idempotency_key, state, attempts, next_attempt_at, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, 'pending', 0, ?, ?, ?)
(id, transport, destination, user_id, job_id, payload_json, idempotency_key, state, attempts, next_attempt_at, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, 'pending', 0, ?, ?, ?)
ON CONFLICT(transport, idempotency_key) DO NOTHING`,
).run(...clean([
`ntf_${randomBytes(8).toString('hex')}`,
input.transport,
input.destination.slice(0, 500),
input.userId ?? null,
input.jobId ?? null,
JSON.stringify(input.payload),
input.idempotencyKey.slice(0, 300),
now,
Expand All @@ -5914,6 +5936,16 @@ export class Store {
return this.get<NotificationOutboxRow>('SELECT * FROM notification_outbox WHERE id = ?', id)
}

// #544: every send a job produced, newest first. The console asks this for the
// job an occurrence dispatched; an empty answer is a run that queued nothing,
// which is `none` on the page rather than an unread `unknown`.
notificationsForJob(jobId: string): NotificationOutboxRow[] {
return this.all<NotificationOutboxRow>(
'SELECT * FROM notification_outbox WHERE job_id = ? ORDER BY created_at DESC, rowid DESC',
jobId,
)
}

// Claiming is one transaction, like the occurrence lease: the due rows are
// read and taken under the same lock, so two workers cannot both hold one row.
// Only `pending` and a due `retry` are claimable. `unknown` is deliberately
Expand Down
13 changes: 8 additions & 5 deletions src/jobs/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1752,7 +1752,7 @@ export class Jobs {
issueUrl: issueUrl,
summary: redact(result.summary ?? job.summary ?? ''),
error: redact(result.error ?? job.error ?? ''),
}, `job.${job.id}:${result.status}:webhook`)
}, `job.${job.id}:${result.status}:webhook`, job.id)
}
const repo = this.store.getRepo(job.repo_id)
if (repo !== undefined && repo.job_budget_cny > 0) {
Expand Down Expand Up @@ -1864,7 +1864,7 @@ export class Jobs {
jobId,
recurrenceKey: record.recurrence.recurrence_key,
count: record.recurrence.count,
}, `postmortem.recurring:${record.postmortem.repo_id}:${record.recurrence.recurrence_key}:${record.recurrence.count}`)
}, `postmortem.recurring:${record.postmortem.repo_id}:${record.recurrence.recurrence_key}:${record.recurrence.count}`, jobId)
} catch (error) {
log.warn('postmortem write failed', { trace: jobId, error: String(error) })
// #666: a postmortem that was not written is a repair job, not a no-op, so
Expand Down Expand Up @@ -1991,7 +1991,7 @@ export class Jobs {
this.store.setSetting(key, nowIso())
this.store.notify(job.user_id, 'job', `${label} 连续 3 个任务失败,自动迭代已暂停;请检查 bridge、GitHub 凭据和模型额度后手动重新开启。`)
this.store.audit(job.user_id, 'ops.repeated_failures', `${label}: 3 consecutive failures`)
this.queueWebhook('repo.repeated_failures', { text: `${label}: 3 consecutive job failures; auto-iterate paused`, repoId: job.repo_id, jobId: job.id }, `repo.repeated_failures:${job.repo_id}:${job.id}`)
this.queueWebhook('repo.repeated_failures', { text: `${label}: 3 consecutive job failures; auto-iterate paused`, repoId: job.repo_id, jobId: job.id }, `repo.repeated_failures:${job.repo_id}:${job.id}`, job.id)
}

private async refreshRepoAccess(repo: RepoRow): Promise<RepoRow | undefined> {
Expand Down Expand Up @@ -2826,7 +2826,7 @@ export class Jobs {
// webhook call; the settings keys above are what make the pass idempotent.
private notifyPlan(job: JobRow, message: string, code: string): void {
this.store.notify(job.user_id, 'job', message, code)
this.queueWebhook(code, { text: message, repoId: job.repo_id, jobId: job.id }, `${code}:${job.id}`)
this.queueWebhook(code, { text: message, repoId: job.repo_id, jobId: job.id }, `${code}:${job.id}`, job.id)
}

private async releaseAllPendingFollowups(): Promise<void> {
Expand Down Expand Up @@ -3321,12 +3321,15 @@ export class Jobs {
// logical event, so a replayed callback or a second tick keeps one row instead
// of sending twice. With no webhook configured there is no destination to
// record, which is the same no-op the old fire-and-forget call had.
private queueWebhook(event: string, payload: Record<string, unknown>, key: string): void {
private queueWebhook(event: string, payload: Record<string, unknown>, key: string, jobId?: string | null): void {
if (this.config.notifyWebhookUrl === '') return
try {
this.store.enqueueNotification({
transport: 'webhook',
destination: 'webhook',
// #544: a send about a job names it, so a scheduled run's delivery is
// readable without parsing the key.
...(jobId === undefined ? {} : { jobId }),
payload: { event, ...payload },
idempotencyKey: key,
})
Expand Down
80 changes: 74 additions & 6 deletions src/schedules/console.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,68 @@
// formula.
//
// Every value it cannot know is `null`, and the page prints `unknown`. A schedule
// with no occurrence row has not run yet, which is not "zero runs"; a delivery has
// no outbox in this slice, so its result is `unknown` rather than a happy zero.
// with no occurrence row has not run yet, which is not "zero runs"; a run whose
// job has finished and queued no notification is `none`, which is also not a
// happy zero; a run whose job is still queued or running is `unknown`, because
// a job queues its notifications only when it ends.

import type { ScheduleOccurrenceRow, ScheduleRow } from '../db.js'
import type { NotificationOutboxState, ScheduleOccurrenceRow, ScheduleRow } from '../db.js'
import { nextTime, type Schedule } from './next.js'
import { scheduleFromRow } from './tick.js'

/**
* What the console can say about the newest run's delivery. `none` is a finished
* run that queued nothing; `unknown` is a caller that read nothing, a schedule
* that never ran, or a run whose job has not finished; `unconfirmed` is an
* outbox row whose send may have crossed the boundary and is never resent
* automatically.
*/
export type ScheduleDelivery = 'unknown' | 'none' | 'unconfirmed' | 'dead' | 'blocked' | 'retry' | 'sending' | 'pending' | 'sent'

// One row per state, worst first: a state that needs a person outranks one that
// needs time, which outranks one that is in flight, which outranks one that is
// already done. Several sends for one job (a webhook that is retrying beside one
// that was accepted) must not read as the accepted one. `dead` outranks
// `blocked` even though both need a person: a dead send spent its retries
// against a destination that exists and is lost without one, while a blocked
// send never left — its destination was never reachable — and the row is still
// intact for a person to fix and resend.
const DELIVERY_RANK: Record<NotificationOutboxState, number> = {
unknown: 0, dead: 1, blocked: 2, retry: 3, sending: 4, pending: 5, sent: 6,
}

function worstDelivery(states: NotificationOutboxState[]): ScheduleDelivery {
let worst = states[0] ?? 'sent'
for (const state of states) if (DELIVERY_RANK[state] < DELIVERY_RANK[worst]) worst = state
return worst === 'unknown' ? 'unconfirmed' : worst
}

function deliveryOf(
newest: ScheduleOccurrenceRow | undefined,
jobStatus: (jobId: string) => string | null,
read: ((jobId: string) => Array<{ state: NotificationOutboxState }>) | undefined,
): ScheduleDelivery {
if (newest === undefined) return 'unknown'
// Only a decided instant can answer. `dispatched` names the job whose
// notifications are the delivery fact, and `skipped` provably queued none.
// A `claimed` row keeps its `job_id` NULL until the worker starts the job,
// and an `expired` row is a lease that lapsed before anyone settled it, so
// neither has decided yet and `none` would be a lie the page prints as
// "not queued".
if (newest.state !== 'dispatched' && newest.state !== 'skipped') return 'unknown'
if (newest.job_id === null) return newest.state === 'skipped' ? 'none' : 'unknown'
if (read === undefined) return 'unknown'
// A job queues its notifications only when it ends (`succeeded`/`failed`),
// so while it is queued or running the empty outbox read is a value nobody
// knows yet — printing `none` there would deny a notification the run may
// still queue. A job the store cannot find is unknown for the same reason:
// the console cannot prove it will never queue one.
const status = jobStatus(newest.job_id)
if (status === null || status === 'queued' || status === 'running') return 'unknown'
const rows = read(newest.job_id)
return rows.length === 0 ? 'none' : worstDelivery(rows.map((row) => row.state))
}

export interface OccurrenceView {
id: string
logicalTime: number
Expand Down Expand Up @@ -52,8 +107,15 @@ export interface ScheduleConsoleState {
last: OccurrenceView | null
/** The newest dispatched instant whose job has succeeded. */
lastSuccess: OccurrenceView | null
/** There is no delivery outbox in this slice, so the result is never known. */
delivery: 'unknown'
/**
* #544: the delivery of the newest decided instant, read through the job it
* dispatched. `unknown` means nobody read the outbox, nothing has run, or the
* run's job has not finished — a job queues its notifications only when it
* ends; `none` means a finished job queued no notification. The outbox's own
* `unknown` — the send may have crossed the external boundary — is reported as
* `unconfirmed` so it is never mistaken for "nobody looked".
*/
delivery: ScheduleDelivery
/**
* #544 slice 6: the schedule layer of the run budget. `dailyRuns` null is
* unbounded; `runsToday` null is unknown, which the page prints as unknown and
Expand Down Expand Up @@ -85,6 +147,11 @@ function viewOf(row: ScheduleOccurrenceRow, now: number): OccurrenceView {
* a success; an unknown job answers null and is not counted as either.
* `runsToday` is `Store.countScheduleDispatchesBetween` over the current UTC day;
* a caller that did not read it leaves it out, and the page says unknown.
* `jobDeliveries` resolves the notifications a job queued; a caller that did not
* read the outbox leaves it out, and the page says unknown rather than `none`.
* `jobStatus` decides the same line: a job still queued or running has not
* queued anything yet, so its empty read is unknown, and only a terminal job
* whose read is empty answers `none`.
*/
export function scheduleConsole(input: {
repoId: string
Expand All @@ -93,6 +160,7 @@ export function scheduleConsole(input: {
now: number
jobStatus: (jobId: string) => string | null
runsToday?: number
jobDeliveries?: (jobId: string) => Array<{ state: NotificationOutboxState }>
}): ScheduleConsoleState {
const { row, now } = input
const schedule = row === undefined ? null : scheduleFromRow(row)
Expand All @@ -116,7 +184,7 @@ export function scheduleConsole(input: {
nextAt: schedule === null ? null : nextTime(schedule, now),
last: occurrences[0] ?? null,
lastSuccess,
delivery: 'unknown',
delivery: deliveryOf(input.occurrences[0], input.jobStatus, input.jobDeliveries),
budget: {
dailyRuns,
runsToday,
Expand Down
Loading
Loading