Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions static/dashboard/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i
(the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this
file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`.

- **1.6.2** - Box logs no longer identify the user, on boxes installed before bake patches 0010-0012 (K2GO-559). At its first start, dash-node applies the same change once: nginx access log without client IP and User-Agent (`/etc/nginx/nginx.conf`), Forgejo router log off (`LOGGER_ROUTER_MODE` empty in `app.ini`), Calibre-Web log level DEBUG -> INFO (`app.db`). A value an admin changed is kept. Each change applies at the service restart (Forgejo: at the next app open, as a pending privacy change; nginx and Calibre-Web: at the next box start). After that restart, the older logs of each changed service are removed once (rotated copies deleted, live log emptied). Markers: `/library/dashboard/log-less.done`, `/library/dashboard/log-less-purge.done`. No REST change. (K2GO-559)
- **1.6.1** - The box restarts conex-core (Content Exchange) on its own when its process is gone: service-heal probes `/v1/apps` and runs `pdsm restart conex-core` only on a refused connection (a busy, slow service is never restarted mid-operation), and skips boxes without the conex-core pdsm service. `POST /system/service/conex-core/restart` is allowed. Localhost-only. (K2GO-574)
- **1.6.0** - Privacy switches for the box services that contact an outside server on their own (K2GO-559). `GET /system/privacy` reports each one (Kolibri statistics ping, Forgejo update check): installed, and whether it may contact its server. `POST /system/privacy/:service {outbound}` sets the service's own config key (Kolibri `[Deployment] DISABLE_PING` in `options.ini`, Forgejo `[cron.update_checker] ENABLED` in `app.ini`) and only writes the config; it never restarts a service. `pending` reports a change that waits for a restart (config newer than the running process; derived on each read, not stored). The app asks the user (Restart now / Later) and requests the restart through `POST /system/service/:svc/restart` (ADR-5343), which now runs one restart per service at a time; a pending change is restarted at the next app open. A missing key reads as the service default (on). On its first start a box sets every installed switch OFF once (new bakes ship OFF), then writes `/library/dashboard/privacy-defaults.done` so a switch the user turns on later stays on; it applies like any pending change. Localhost-only. (K2GO-559)
- **1.5.2** - FQR maps on the new tile-extract.py CLI (iiab#4574, carried as `tools/upstream-patches` 0004) (K2GO-566). The download, the size estimate and the delete follow the CLI of the installed script: `--bbox=`, `--yes`, `--progress json` and `--threads 8` when it has them, else the older positional form (a box can run a newer dash-node than its rootfs). Delete uses `--yes` with no stdin pipe (no EPIPE) and also removes an unfinished download; `POST /maps/delete` returns 409 `{busy: true}` while another FQR download or delete holds the box lock. The script runs one FQR download or delete at a time, so a second download now waits as `queued` and then runs (before, both ran in parallel). A purge after a cancel or a hard error never removes a region that the job did not make. A Cancel of a paused download now removes its unfinished files (new engine hook `registerDiscard`), and a Cancel during a pause's flush wins over the pause. The script's own catalog and mirror network errors and an outside SIGTERM (exit 3) keep the partial (paused) instead of a purge. A download that ends with no region (the script exits 0 because the box overlaps a region that an earlier queued job made) is an error, not `done`. Localhost-only. (K2GO-566)
Expand Down
4 changes: 2 additions & 2 deletions static/dashboard/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions static/dashboard/package.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"name": "dashboard-console",
"version": "1.6.1",
"version": "1.6.2",
"description": "",
"main": "index.js",
"scripts": {
"test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts sockets/log-rotate.test.ts sockets/rebuild-status.test.ts sockets/conex-proxy.test.ts sockets/maps.fqr-behavior.test.ts sockets/privacy.test.ts",
"test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts sockets/services.test.ts sockets/log-rotate.test.ts sockets/rebuild-status.test.ts sockets/conex-proxy.test.ts sockets/maps.fqr-behavior.test.ts sockets/privacy.test.ts sockets/log-less.test.ts",
"test:db": "node --require ts-node/register --test sockets/jobs.test.ts",
"typecheck": "tsc --noEmit",
"build": "tsc",
Expand Down
9 changes: 9 additions & 0 deletions static/dashboard/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { apiRouter } from './routes';
import { startServiceHeal } from './sockets/service-heal';
import { startLogRotation, stopLogRotation } from './sockets/log-rotate';
import { applyPrivacyDefaultsOnce, PRIVACY_DEFAULTS_MARKER, PRIVACY_SWITCHES } from './sockets/privacy';
import { applyLogLessOnce, LOG_LESS_MARKER, LOG_LESS_TARGETS, LOG_PURGE_MARKER, purgeOldLogsOnce } from './sockets/log-less';

// K2GO-559: one-time privacy defaults, before the API answers. Only the live service runs it: the
// rebuild's staged smoke instance sets PORT and must not change the box (tools/rebuild-dashboard.sh).
Expand All @@ -27,6 +28,14 @@ if (process.env.PORT === undefined) {
} catch (e) {
console.error('[privacy] one-time defaults failed; retry at next start', e);
}
try {
const r = applyLogLessOnce(LOG_LESS_TARGETS, LOG_LESS_MARKER);
if (r.ran) console.log(`[log-less] applied once; changed (applies at the service restart): ${r.changed.join(', ') || 'none'}`);
const p = purgeOldLogsOnce(LOG_LESS_MARKER, LOG_PURGE_MARKER);
if (p.ran) console.log(`[log-less] old logs removed: ${p.purged.join(', ') || 'none'}; waiting for a restart: ${p.waiting.join(', ') || 'none'}`);
} catch (e) {
console.error('[log-less] one-time step failed; retry at next start', e);
}
}

const app = express();
Expand Down
160 changes: 160 additions & 0 deletions static/dashboard/sockets/log-less.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
/// <reference types="node" />
import test from 'node:test';
import assert from 'node:assert/strict';
import fs from 'fs';
import os from 'os';
import path from 'path';
import {
applyLogLessOnce, forgejoWithoutRouterLog, nginxWithoutClientIds, purgeOldLogsOnce, rotatedBefore,
LogLessStep, ServiceLog,
} from './log-less';

const STOCK_NGINX = [
'http {',
' log_format awstats',
` '$remote_addr - $remote_user [$time_local] "$request" '`,
` '$status $body_bytes_sent "$http_referer" '`,
` '"$http_user_agent" "http_x_forwarded_for"';`,
'',
' access_log /var/log/nginx/access.log awstats;',
` log_format scripts '$request > $document_root$fastcgi_script_name $fastcgi_path_info';`,
'}',
'',
].join('\n');

test('nginx: the stock awstats format loses the client IP and User-Agent, same fields', () => {
const out = nginxWithoutClientIds(STOCK_NGINX)!;
assert.ok(out.includes(`'- - $remote_user [$time_local] "$request" '`));
assert.ok(out.includes(`'"-" "http_x_forwarded_for"';`));
assert.ok(!out.includes('$remote_addr') && !out.includes('$http_user_agent'));
assert.ok(out.includes(`log_format scripts '$request >`)); // other formats stay
assert.equal(nginxWithoutClientIds(out), undefined); // second run: nothing to do
});

test('nginx: a changed or missing awstats format is left alone', () => {
assert.equal(nginxWithoutClientIds('http {\n access_log off;\n}\n'), undefined);
assert.equal(nginxWithoutClientIds(`log_format awstats '$remote_user "$request"';`), undefined);
});

test('forgejo: the router log is turned off in [log]; a key the admin set is kept', () => {
const t = '[mailer]\nENABLED = false\n\n[log]\nROOT_PATH = /var/log/forgejo\nMODE = console\nLEVEL = Info\n';
const out = forgejoWithoutRouterLog(t)!;
assert.ok(/\[log\][^[]*LOGGER_ROUTER_MODE = \s*\n/.test(out));
assert.equal(forgejoWithoutRouterLog(out), undefined);
assert.equal(forgejoWithoutRouterLog('[log]\nDISABLE_ROUTER_LOG = false\n'), undefined);
assert.equal(forgejoWithoutRouterLog('[log]\nlogger.router.MODE = console\n'), undefined);
});

function box(files: { nginx?: string; forgejo?: string; calibre?: boolean }) {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'logless-'));
const t = {
nginxConf: path.join(dir, 'nginx.conf'),
forgejoIni: path.join(dir, 'app.ini'),
calibreAppDb: path.join(dir, 'app.db'),
};
if (files.nginx !== undefined) fs.writeFileSync(t.nginxConf, files.nginx);
if (files.forgejo !== undefined) fs.writeFileSync(t.forgejoIni, files.forgejo);
if (files.calibre) fs.writeFileSync(t.calibreAppDb, '');
return { t, marker: path.join(dir, 'log-less.done') };
}

test('once: first run changes every installed service, then writes the marker; later runs do nothing', () => {
const b = box({ nginx: STOCK_NGINX, forgejo: '[log]\nMODE = console\n', calibre: true });
const calls: string[] = [];
const calibre = (f: string) => { calls.push(f); return true; };
assert.deepEqual(applyLogLessOnce(b.t, b.marker, calibre), { ran: true, changed: ['nginx', 'forgejo', 'calibre-web'] });
assert.deepEqual(calls, [b.t.calibreAppDb]);
assert.ok(!fs.readFileSync(b.t.nginxConf, 'utf8').includes('$remote_addr'));
assert.equal(fs.existsSync(b.marker), true);
assert.deepEqual(applyLogLessOnce(b.t, b.marker, calibre), { ran: false, changed: [] });
assert.equal(calls.length, 1);
});

test('once: services that are not installed are skipped', () => {
const b = box({});
assert.deepEqual(applyLogLessOnce(b.t, b.marker, () => { throw new Error('not called'); }), { ran: true, changed: [] });
assert.equal(fs.existsSync(b.t.forgejoIni), false);
});

test('once: a failed step leaves no marker, so the next start tries again', () => {
const b = box({ nginx: STOCK_NGINX, calibre: true });
assert.throws(() => applyLogLessOnce(b.t, b.marker, () => { throw new Error('database is locked'); }));
assert.equal(fs.existsSync(b.marker), false);
assert.deepEqual(applyLogLessOnce(b.t, b.marker, () => false), { ran: true, changed: [] }); // nginx already done
});

test('once: the marker records which services changed (input of the purge)', () => {
const b = box({ nginx: STOCK_NGINX });
applyLogLessOnce(b.t, b.marker, () => false);
assert.deepEqual(JSON.parse(fs.readFileSync(b.marker, 'utf8')).changed, ['nginx']);
});

// --- Purge of the logs written before the change ---------------------------------------------------

function purgeBox(changed: LogLessStep[]) {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'logpurge-'));
const marker = path.join(dir, 'log-less.done');
const old = Date.now() / 1000 - 3600; // one hour before the change
const logs = {} as Record<LogLessStep, ServiceLog>;
for (const svc of ['nginx', 'forgejo', 'calibre-web'] as LogLessStep[]) {
const log = path.join(dir, `${svc}.log`);
fs.writeFileSync(log, '192.168.1.20 - - "GET /" "Mozilla/5.0 (Linux; moto g)"\n');
for (const r of [`${log}.1`, `${log}.2.gz`]) { fs.writeFileSync(r, 'old'); fs.utimesSync(r, old, old); }
logs[svc] = { log, process: svc };
}
fs.writeFileSync(path.join(dir, 'nginx.log.bak'), 'not a rotated copy');
fs.utimesSync(path.join(dir, 'nginx.log.bak'), old, old);
fs.writeFileSync(marker, JSON.stringify({ at: 'x', changed }) + '\n');
return { dir, marker, purgeMarker: path.join(dir, 'log-less-purge.done'), logs };
}

test('rotatedBefore: only <log>.N and <log>.N.gz older than the change', () => {
const b = purgeBox([]);
const got = rotatedBefore(b.logs.nginx.log, Date.now()).map((f) => path.basename(f)).sort();
assert.deepEqual(got, ['nginx.log.1', 'nginx.log.2.gz']);
assert.deepEqual(rotatedBefore(b.logs.nginx.log, Date.now() - 2 * 3600_000), []);
});

test('purge: a changed service on the new config loses its old logs; others are not touched', () => {
const b = purgeBox(['nginx', 'calibre-web']);
const started = Date.now() + 1000; // restarted after the change
const r = purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, () => started);
assert.deepEqual(r, { ran: true, purged: ['nginx', 'calibre-web'], waiting: [] });
assert.equal(fs.statSync(b.logs.nginx.log).size, 0);
assert.equal(fs.existsSync(`${b.logs.nginx.log}.1`), false);
assert.equal(fs.existsSync(`${b.logs.nginx.log}.2.gz`), false);
assert.equal(fs.existsSync(path.join(b.dir, 'nginx.log.bak')), true);
assert.ok(fs.statSync(b.logs.forgejo.log).size > 0); // not changed: no purge
assert.equal(fs.existsSync(`${b.logs.forgejo.log}.1`), true);
assert.equal(fs.existsSync(b.purgeMarker), true);
assert.equal(purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, () => started).ran, false);
});

test('purge: a service still on the old config waits; the next start purges it', () => {
const b = purgeBox(['nginx', 'forgejo']);
const startOf = (p: string) => (p === 'forgejo' ? Date.now() - 3600_000 : null); // nginx not running
assert.deepEqual(purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, startOf),
{ ran: true, purged: ['nginx'], waiting: ['forgejo'] });
assert.ok(fs.statSync(b.logs.forgejo.log).size > 0);
assert.equal(fs.existsSync(b.purgeMarker), false);
assert.deepEqual(purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, () => Date.now() + 1000),
{ ran: true, purged: ['nginx', 'forgejo'], waiting: [] });
assert.equal(fs.statSync(b.logs.forgejo.log).size, 0);
assert.equal(fs.existsSync(b.purgeMarker), true);
});

test('purge: a copy rotated after the change but before the restart is removed too', () => {
const b = purgeBox(['nginx']);
const rotatedLater = `${b.logs.nginx.log}.3`;
fs.writeFileSync(rotatedLater, '10.0.0.7 - - "GET /" "Mozilla/5.0"'); // mtime now: after the marker
const r = purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, () => Date.now() + 1000);
assert.deepEqual(r.purged, ['nginx']);
assert.equal(fs.existsSync(rotatedLater), false);
});

test('purge: nothing before the log-less step, nothing to do when it changed nothing', () => {
const b = purgeBox([]);
assert.equal(purgeOldLogsOnce(path.join(b.dir, 'missing'), b.purgeMarker, b.logs, () => null).ran, false);
assert.deepEqual(purgeOldLogsOnce(b.marker, b.purgeMarker, b.logs, () => null), { ran: true, purged: [], waiting: [] });
assert.ok(fs.statSync(b.logs.nginx.log).size > 0);
});
Loading
Loading