Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,26 @@ ARTIFACT_RETENTION_KEEP=3
ARTIFACT_RETENTION_MAX_AGE_DAYS=14
ARTIFACT_RETENTION_ROOT=

# ── Data retention (#281) ────────────────────────────────────────────────
# Age windows for the scheduler's prune pass, in days. Unset takes the built-in
# value shown after the name, and that is what the service deletes by today: an
# empty line never means "keep forever". A value that is not an integer or is
# below 1 falls back to the default rather than becoming 0 ("delete everything").
# Preview a change first: node scripts/retention.mjs --dry-run --days N
# RETENTION_OAUTH_STATES_DAYS=7
# RETENTION_APPROVAL_TOKENS_DAYS=30
# RETENTION_NOTIFICATIONS_READ_DAYS=30
# RETENTION_NOTIFICATIONS_UNREAD_DAYS=90
# RETENTION_EVENTS_DAYS=90
# RETENTION_JOB_EVENTS_DAYS=365
# RETENTION_WEBHOOK_DELIVERIES_DAYS=30
# RETENTION_OUTBOX_SENT_DAYS=30
# RETENTION_OUTBOX_DEAD_DAYS=90
# RETENTION_OUTBOX_UNKNOWN_DAYS=365
# RETENTION_JOBS_DAYS=365
# RETENTION_AUDIT_LOG_DAYS=365
# RETENTION_QUOTA_LEDGER_DAYS=730

# ── Storage ──────────────────────────────────────────────────────────────
DATA_DIR=./data
# Optional; defaults to DATA_DIR/backups.
Expand Down
1 change: 1 addition & 0 deletions changelog.d/281.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Operators can set each data-retention window with a `RETENTION_*_DAYS` environment variable and preview a change before making it: the new read-only `node scripts/retention.mjs --dry-run --days N` prints, per table, how many rows that window would delete without deleting anything, and the `/admin` page gains a **Data retention** card with the effective window per class and its current count. A window below one day is refused by the delete path as well as by the config, so a direct `pruneOldData({ jobs: 0 })` throws instead of deleting every finished job.
22 changes: 22 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -2254,6 +2254,28 @@ all, so the console can report it only as staleness plus a mismatch; the drill
prints its verdict and records nothing, so the card says "restore drill: none
recorded" rather than a green it cannot prove.

### Data retention windows (#281)

`Store.pruneOldData` is the one age-based delete pass, run from the scheduler
tick (at most hourly) and otherwise unchanged. Every window now comes from
`config.retention.windows`, built in `loadConfig` from `scripts/lib/retention.mjs`
— one table that also feeds each predicate and the preview, so the rows a preview
counts are exactly the rows a deploy deletes. Unset keeps the literal this
service has always used (OAuth states 7 days; approval tokens, read
notifications, webhook deliveries and sent outbox 30; unread notifications and
events 90; dead/blocked outbox 90; job events, finished jobs, audit log and
unknown outbox 365; quota ledger 730), and `configured` is false for a fallback
so the page can mark it `default`. Unset never means "keep forever": that would
silently stop deleting rows the service deletes today. `sessions` are removed by
`expires_at` and `api_tokens` have no age window; both are reported as such
rather than given a number. `Store.retentionCounts` counts at the configured
windows for the `/admin` card, which is read-only and has no window control; the
operator tool `scripts/retention.mjs --dry-run --days N` counts any window on a
host copy. The delete path refuses a window that is not a whole number of days of
at least 1, before the first delete: `loadConfig` already refuses such an env
value, and `pruneOldData({ jobs: 0 })` is a direct caller the config cannot see,
so the store checks the same invariant rather than trusting its input.

### Operational alerts

A failed job also feeds the ops signal: three consecutive failures in one
Expand Down
48 changes: 48 additions & 0 deletions docs/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -1472,6 +1472,54 @@ Run it from a timer only after a dry run has shown the candidates:
10 4 * * * cd /opt/fledge && node scripts/artifact-retention.mjs --root /var/lib/fledge/flybot-users --apply
```

### Data retention windows

`pruneOldData` runs in the scheduler tick (at most hourly) and deletes rows older
than a per-class window. Each window is an environment variable; unset takes the
built-in value, which is the number the service already enforced:

| Variable | Class | Default |
|---|---|---|
| `RETENTION_OAUTH_STATES_DAYS` | `oauth_states` | 7 |
| `RETENTION_APPROVAL_TOKENS_DAYS` | `approval_tokens` (expired or used) | 30 |
| `RETENTION_NOTIFICATIONS_READ_DAYS` | read `notifications` | 30 |
| `RETENTION_NOTIFICATIONS_UNREAD_DAYS` | unread `notifications` | 90 |
| `RETENTION_EVENTS_DAYS` | `events` | 90 |
| `RETENTION_JOB_EVENTS_DAYS` | `job_events` | 365 |
| `RETENTION_WEBHOOK_DELIVERIES_DAYS` | `webhook_deliveries` | 30 |
| `RETENTION_OUTBOX_SENT_DAYS` | sent `notification_outbox` | 30 |
| `RETENTION_OUTBOX_DEAD_DAYS` | dead/blocked `notification_outbox` | 90 |
| `RETENTION_OUTBOX_UNKNOWN_DAYS` | unknown `notification_outbox` | 365 |
| `RETENTION_JOBS_DAYS` | finished `jobs` | 365 |
| `RETENTION_AUDIT_LOG_DAYS` | `audit_log` | 365 |
| `RETENTION_QUOTA_LEDGER_DAYS` | `quota_ledger` | 730 |

An unset window is the default, never "keep forever", and a value that is not an
integer or is below 1 falls back to the default rather than being read as zero
(zero here would mean "delete every row"). `sessions` are removed by
`expires_at` and `api_tokens` have no age window; neither is configurable here.

The store refuses as well as the config. `Store.pruneOldData` throws on a window
that is not a whole number of days of at least 1 — including a caller that passes
`{ jobs: 0 }` directly, which without the guard would delete every finished job —
and it refuses before the first delete, so a call that cannot be honoured removes
nothing.

Preview any change before making it. The tool opens the database read-only, holds
no write lock, prints one `count=` line per class and two states instead of a
number:

```sh
sudo -u fledge DB_PATH=/var/lib/fledge/fledge.db node scripts/retention.mjs --dry-run --days 30
```

`--days` is required; it never guesses a window. `--dry-run` is accepted for
symmetry with `npm run artifacts:prune`, and the tool never deletes rows with or
without it. The output is counts only — never a row body, login, email or token
value — so the preview can be pasted into a log. The same counts for the
configured windows appear on `/admin`'s **Data retention** card, with an unset
window marked `default`.

## 10. Go-live checklist

- [ ] HTTPS `BASE_URL`; OAuth callback matches exactly.
Expand Down
15 changes: 15 additions & 0 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -619,6 +619,21 @@ whose duplicate risk a person still has to judge; a human resend is audited as
deletes the GitHub installation row with the account; it does not call GitHub
to uninstall the App installation.

- **Row retention windows (#281).** The scheduler's age pass deletes
`oauth_states`, `approval_tokens`, read and unread `notifications`, `events`,
`job_events`, `webhook_deliveries`, the three `notification_outbox` states,
finished `jobs`, `audit_log` and `quota_ledger` by a per-class window; the
numbers are configuration (`RETENTION_*_DAYS`) with the shipped literal as the
effective default. An unset window is never "keep forever": that would stop a
deletion the service performs today. A value that is not an integer or is
below 1 falls back to the default rather than becoming zero, because zero here
would mean "delete every row". `sessions` age out by `expires_at` and
`api_tokens` are revoked or removed with their account; the preview names both
states instead of printing a count. Before a window is shortened,
`node scripts/retention.mjs --dry-run --days N` counts what it would delete —
read-only, printing counts and never a row body, login, email or token value.
The `/admin` card shows the same counts for the configured windows.

### Platform emergency stop

- `POST /admin/platform/stop` and `POST /admin/platform/resume` are the
Expand Down
57 changes: 57 additions & 0 deletions scripts/lib/retention.d.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
export type RetentionId =
| 'oauthStates'
| 'approvalTokens'
| 'notificationsRead'
| 'notificationsUnread'
| 'events'
| 'jobEvents'
| 'webhookDeliveries'
| 'outboxSent'
| 'outboxDead'
| 'outboxUnknown'
| 'jobs'
| 'auditLog'
| 'quotaLedger'

export type RetentionWindows = Record<RetentionId, number>

export interface RetentionAt {
now: string
cutoff: string
}

export interface RetentionWhere {
sql: string
params: string[]
}

export interface RetentionClass {
id: RetentionId
key: string
table: string
env: string
defaultDays: number
label: readonly [string, string]
where: (at: RetentionAt) => RetentionWhere
}

export interface RetentionIdleTable {
id: 'sessions' | 'api_tokens'
state: 'expiry-based' | 'no-age-window'
label: readonly [string, string]
}

export interface RetentionCount {
id: RetentionId
key: string
days: number
count: number
}

export const RETENTION_CLASSES: readonly RetentionClass[]
export const RETENTION_IDLE_TABLES: readonly RetentionIdleTable[]
export function retentionDefaultWindows(): RetentionWindows
export function retentionWindow(raw: string | undefined | null, fallback: number): { days: number; configured: boolean }
export function retentionDeleteSql(cls: RetentionClass, at: RetentionAt): { sql: string; params: string[] }
export function retentionCountSql(cls: RetentionClass, at: RetentionAt): { sql: string; params: string[] }
export function retentionCountsAt(count: (sql: string, params: string[]) => number, days: number, now?: number): RetentionCount[]
195 changes: 195 additions & 0 deletions scripts/lib/retention.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
// #281: the age windows `pruneOldData` enforces, in one table.
//
// The store, the read-only dry-run tool and the admin page all read this file.
// A window that can delete rows and a window that a preview reports must be the
// same number and the same predicate, or the preview is a guess; keeping them in
// one place is what makes `scripts/retention.mjs --days N` honest about what a
// deploy will do.
//
// `defaultDays` is the literal this service shipped with, and it is also the
// effective value when the env var is unset: "unset" must never mean "keep
// forever", because that would silently stop deleting rows that are deleted
// today.

/**
* One retention class. `where` returns the SQL after `WHERE` plus its params;
* it receives the run instant and the cutoff so a predicate that compares
* against "now" (an expired approval token) cannot accidentally compare against
* the cutoff instead.
*/
export const RETENTION_CLASSES = [
{
id: 'oauthStates',
key: 'oauth_states',
table: 'oauth_states',
env: 'RETENTION_OAUTH_STATES_DAYS',
defaultDays: 7,
label: ['OAuth 状态', 'OAuth states'],
where: ({ cutoff }) => ({ sql: 'created_at < ?', params: [cutoff] }),
},
{
id: 'approvalTokens',
key: 'approval_tokens',
table: 'approval_tokens',
env: 'RETENTION_APPROVAL_TOKENS_DAYS',
defaultDays: 30,
label: ['审批令牌', 'Approval tokens'],
where: ({ now, cutoff }) => ({ sql: '(expires_at < ? OR used_at IS NOT NULL) AND created_at < ?', params: [now, cutoff] }),
},
{
id: 'notificationsRead',
key: 'notifications_read',
table: 'notifications',
env: 'RETENTION_NOTIFICATIONS_READ_DAYS',
defaultDays: 30,
label: ['已读通知', 'Read notifications'],
where: ({ cutoff }) => ({ sql: 'read_at IS NOT NULL AND created_at < ?', params: [cutoff] }),
},
{
id: 'notificationsUnread',
key: 'notifications_unread',
table: 'notifications',
env: 'RETENTION_NOTIFICATIONS_UNREAD_DAYS',
defaultDays: 90,
label: ['未读通知', 'Unread notifications'],
where: ({ cutoff }) => ({ sql: 'read_at IS NULL AND created_at < ?', params: [cutoff] }),
},
{
id: 'events',
key: 'events',
table: 'events',
env: 'RETENTION_EVENTS_DAYS',
defaultDays: 90,
label: ['事件', 'Events'],
where: ({ cutoff }) => ({ sql: 'at < ?', params: [cutoff] }),
},
{
id: 'jobEvents',
key: 'job_events',
table: 'job_events',
env: 'RETENTION_JOB_EVENTS_DAYS',
defaultDays: 365,
label: ['任务事件', 'Job events'],
where: ({ cutoff }) => ({ sql: 'at < ?', params: [cutoff] }),
},
{
id: 'webhookDeliveries',
key: 'webhook_deliveries',
table: 'webhook_deliveries',
env: 'RETENTION_WEBHOOK_DELIVERIES_DAYS',
defaultDays: 30,
label: ['入站 Webhook 投递', 'Inbound webhook deliveries'],
where: ({ cutoff }) => ({ sql: 'received_at < ?', params: [cutoff] }),
},
// #544 slice 7: a delivered notification is a receipt, so it goes with the
// other 30-day receipts. The unresolved states are kept longer because they
// are what a human still has to look at; `unknown` keeps its full window so
// the duplicate risk stays visible instead of ageing out with the sent rows.
{
id: 'outboxSent',
key: 'outbox_sent',
table: 'notification_outbox',
env: 'RETENTION_OUTBOX_SENT_DAYS',
defaultDays: 30,
label: ['发件箱 · 已发送', 'Outbox · sent'],
where: ({ cutoff }) => ({ sql: "state = 'sent' AND created_at < ?", params: [cutoff] }),
},
{
id: 'outboxDead',
key: 'outbox_dead',
table: 'notification_outbox',
env: 'RETENTION_OUTBOX_DEAD_DAYS',
defaultDays: 90,
label: ['发件箱 · 受阻', 'Outbox · dead or blocked'],
where: ({ cutoff }) => ({ sql: "state IN ('dead', 'blocked') AND created_at < ?", params: [cutoff] }),
},
{
id: 'outboxUnknown',
key: 'outbox_unknown',
table: 'notification_outbox',
env: 'RETENTION_OUTBOX_UNKNOWN_DAYS',
defaultDays: 365,
label: ['发件箱 · 结果未知', 'Outbox · unknown'],
where: ({ cutoff }) => ({ sql: "state = 'unknown' AND created_at < ?", params: [cutoff] }),
},
{
id: 'jobs',
key: 'jobs',
table: 'jobs',
env: 'RETENTION_JOBS_DAYS',
defaultDays: 365,
label: ['任务', 'Jobs'],
where: ({ cutoff }) => ({ sql: "status IN ('succeeded', 'failed', 'quota_exhausted') AND finished_at IS NOT NULL AND finished_at < ?", params: [cutoff] }),
},
{
id: 'auditLog',
key: 'audit_log',
table: 'audit_log',
env: 'RETENTION_AUDIT_LOG_DAYS',
defaultDays: 365,
label: ['审计日志', 'Audit log'],
where: ({ cutoff }) => ({ sql: 'at < ?', params: [cutoff] }),
},
{
id: 'quotaLedger',
key: 'quota_ledger',
table: 'quota_ledger',
env: 'RETENTION_QUOTA_LEDGER_DAYS',
defaultDays: 730,
label: ['额度流水', 'Quota ledger'],
where: ({ cutoff }) => ({ sql: 'at < ?', params: [cutoff] }),
},
]

// These two have no age window at all, and reporting a fabricated number for
// them would be worse than saying so: sessions are reclaimed by `expires_at`
// and `api_tokens` are revoked, or removed by the account cascade. Adding a
// delete path for either is a behaviour change, not a config value.
export const RETENTION_IDLE_TABLES = [
{ id: 'sessions', state: 'expiry-based', label: ['会话', 'Sessions'] },
{ id: 'api_tokens', state: 'no-age-window', label: ['API 令牌', 'API tokens'] },
]

/** Every window at its built-in value. The defaults must equal today's literals. */
export function retentionDefaultWindows() {
return Object.fromEntries(RETENTION_CLASSES.map((cls) => [cls.id, cls.defaultDays]))
}

/**
* Parse one configured window. Empty, non-numeric or below one day means the
* built-in value, and `configured` false is what the operator surface marks as
* `default`. A window of zero is refused rather than read as "delete
* everything": that is a data-loss instruction, not a retention window.
*/
export function retentionWindow(raw, fallback) {
const text = raw === undefined || raw === null ? '' : String(raw).trim()
if (text === '') return { days: fallback, configured: false }
const n = Number(text)
if (!Number.isInteger(n) || n < 1) return { days: fallback, configured: false }
return { days: n, configured: true }
}

/** The DELETE the scheduler runs for one class. */
export function retentionDeleteSql(cls, at) {
const { sql, params } = cls.where(at)
return { sql: `DELETE FROM ${cls.table} WHERE ${sql}`, params }
}

/** The matching COUNT, so a preview cannot select different rows than the delete. */
export function retentionCountSql(cls, at) {
const { sql, params } = cls.where(at)
return { sql: `SELECT COUNT(*) AS n FROM ${cls.table} WHERE ${sql}`, params }
}

/**
* Count what each window would delete at `days`, through the same predicates.
* `count(sql, params)` returns the scalar, so the store and the plain-node tool
* share this loop instead of each keeping its own.
*/
export function retentionCountsAt(count, days, now = Date.now()) {
const at = { now: new Date(now).toISOString(), cutoff: new Date(now - days * 86_400_000).toISOString() }
return RETENTION_CLASSES.map((cls) => {
const { sql, params } = retentionCountSql(cls, at)
return { id: cls.id, key: cls.key, days, count: Number(count(sql, params)) }
})
}
Loading
Loading