Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Authentication uses `Authorization: Bearer <JWT>` headers. JWTs are issued by th

- **API DB** (`cubedocs-main`): projects, docs (+ `doc_ai_summaries` satellite; `docs.slug` is the doc's custom public-URL segment from the `slug:` frontmatter key - unique per project, normalized + synced by `docOps.syncDocSlug` on every save path incl. collab checkpoints and restores, resolved by `/public/docs/:site/:idOrSlug`), doc_revisions, folders, files, members, site_reports + user_reports (abuse reports: public "Report Site" button → `POST /public/projects/:id/report`, profile-card "Report user" → authenticated `POST /users/:id/report`; both triaged via the admin worker's `/api/reports/{sites,users}` routes + the admin "Reports" page's Sites/Users tabs) - see `packages/api/migrations/`. Bound as `DB` on the API worker.
- **Recycle bin** (API DB, `0062_add_trash.sql`, `lib/trash.ts`, `routes/trash.ts`, `components/RecycleBin.tsx` - shown as the "Recycle bin" section of Site Settings, `/projects/:id/trash` redirects there): deleting a doc/file/folder (app, or `/v1` doc DELETE) **moves its rows out of** `docs`/`files`/`folders` into `trash_items` JSON snapshots (one `trash_entries` row per user-visible entry; a folder bins its whole subtree as one entry, except the home doc, which is moved to the root). Because binned rows are physically gone from the live tables, **no reader needs a "not deleted" filter - don't add one, and don't hard-delete docs/files/folders from a route; call `moveToTrash`**. Cascade-lost satellites that aren't derivable (`doc_shares`, the `docs_fts` row, `site_reports.doc_id`) ride in `extra_json` and come back on restore; R2 objects + `asset_revisions` stay until purge. Restore/list = editor+, delete-forever/empty = admin+, auto-purge after 30 days on the daily cron. Site deletion (api + admin) must include `trash_items` ids in its R2 sweep. `trash.test.ts` runs against real SQLite via `src/test/sqliteD1.ts` (all migrations applied) - use that harness for SQL-heavy logic.
- **Doc search index** (API DB, `0064_rebuild_docs_fts.sql`, `lib/fts.ts`): `docs_fts` is FTS5 with an indexed `pid` = `hex(project_id)` token column plus a `docs_fts_map(rid, doc_id)` rowid map, so per-doc writes are rowid lookups and search MATCHes `pid:"<token>"` (`siteFtsMatch`) instead of scanning every site. **Never write `docs_fts` with raw SQL (`WHERE doc_id = ?` / `WHERE project_id = ?` are full scans, and a bare INSERT gets no `pid` or map row) - use the `fts*Statements` helpers**, which keep exactly one map row per FTS row and compose into a caller's batch (the insert + map statements must stay adjacent: `last_insert_rowid()`). Search weights stay `bm25(docs_fts, 5.0, 1.0, 0.0)` (pid weighted 0). `scripts/repair-docs-fts.sql` reconciles rows written by pre-0064 code.
- **Admin account deletion is always delayed** (auth `0034_add_user_deletions.sql`, admin `lib/userDeletion.ts`, Users page checkboxes → bulk Delete/Disable/Re-enable/Cancel deletion): "Delete" only **schedules** - it disables the account now and writes a `user_deletions` row; the admin worker's hourly cron (`runDueUserDeletions`) performs the delete after 7 days, and any admin can cancel before then (restoring the prior moderation). Owned sites/orgs are deleted **only** when the operator ticked an explicit per-account box in the preview, scheduling refuses unless the confirmed ids exactly match current ownership (`sameIdSet`), and execution refuses if the account owns anything outside them. While a deletion is pending, `PATCH /api/users/:id` moderation is refused - cancel instead (the guard is inside the write, not just a pre-check). The cron claims each schedule by `(user_id, id)` with a lease; cancel is refused while leased or once `started_at` is set (first irreversible step), after which a failed run is retried to completion - a confirmed site/org that no longer exists counts as done. Real-SQLite scenario tests: `userDeletion.sqlite.test.ts`. **Don't add an immediate admin delete path.** Site deletion itself is shared via `lib/projectDelete.ts`.
- **Auth DB** (`cubedocs-auth`): users, sessions, totp, webauthn_credentials, plus the 1:1 satellites `user_billing` (Stripe + plan state) and `user_preferences` (fonts, Ink cosmetics, timezone, bio, badges, opt-in profile `handle`) - see `packages/auth/migrations/`. The `handle` (`0035`, partial unique index, stored lowercase, rules in `packages/auth/src/handle.ts` mirrored by `frontend/src/lib/handle.ts`) powers the `/@<handle>` profile link: set/cleared via auth `/update-handle` (proxied as `PATCH /me/handle`), resolved by the API's `GET /users/by-handle/:handle` reading `AUTH_DB` directly. Null = link off. Admins moderate it via the admin worker's `PATCH /api/users/:id/handle` (Users → details → "Profile Link"; may assign reserved names) and can set `handle_locked`, which makes `/update-handle` refuse every user change (403 `reason: "locked"`, guarded inside the write too). Bound as `DB` on the auth worker (read+write), as `AUTH_DB` on the API worker (read-only by convention), and as `AUTH_DB` on the admin worker (read+write for moderation/badges/billing-grants/audit - see the verification-boundary note above). **A schema change to any column these direct readers/writers touch requires redeploying auth + api + admin (in that order).**

Expand Down
6 changes: 5 additions & 1 deletion packages/admin/src/lib/projectDelete.sqlite.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@ function setup() {
for (const [doc, folder] of [[`${p}-d1`, null], [`${p}-d2`, `${p}-f`], [`${p}-d3`, `${p}-sub`]] as const) {
s.prepare("INSERT INTO docs (id, title, project_id, folder_id, author_id, created_at, updated_at) VALUES (?, 'T', ?, ?, 'u-1', ?, ?)")
.run(doc, p, folder, now, now);
s.prepare("INSERT INTO docs_fts (title, body, doc_id, project_id) VALUES ('T', 'body', ?, ?)").run(doc, p);
// Current (0064) FTS layout: site token + rowid map row.
s.prepare("INSERT INTO docs_fts (title, body, pid, doc_id, project_id) VALUES ('T', 'body', hex(?), ?, ?)").run(p, doc, p);
s.prepare("INSERT INTO docs_fts_map (rid, doc_id) VALUES (last_insert_rowid(), ?)").run(doc);
s.prepare("INSERT INTO doc_shares (id, doc_id, user_id, project_id, granted_by, created_at) VALUES (?, ?, 'u-2', ?, 'u-1', ?)")
.run(`${doc}-share`, doc, p, now);
s.prepare("INSERT INTO doc_ai_summaries (doc_id, summary, version) VALUES (?, 'S', 1)").run(doc);
Expand Down Expand Up @@ -52,11 +54,13 @@ describe("deleteProjectCascade (real SQLite)", () => {
expect(count(table, `${col} = 'p-1'`), table).toBe(0);
}
expect(count("doc_ai_summaries", "doc_id LIKE 'p-1-%'")).toBe(0);
expect(count("docs_fts_map", "doc_id LIKE 'p-1-%'")).toBe(0);

// p-2 is untouched.
expect(count("docs", "project_id = 'p-2'")).toBe(3);
expect(count("doc_shares", "project_id = 'p-2'")).toBe(3);
expect(count("docs_fts", "project_id = 'p-2'")).toBe(3);
expect(count("docs_fts_map", "doc_id LIKE 'p-2-%'")).toBe(3);
expect(count("asset_revisions", "project_id = 'p-2'")).toBe(3);
expect(count("doc_ai_summaries", "doc_id LIKE 'p-2-%'")).toBe(3);
expect(count("folders", "project_id = 'p-2'")).toBe(2);
Expand Down
6 changes: 4 additions & 2 deletions packages/admin/src/lib/projectDelete.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { releaseCustomDomain } from "../../../api/src/lib/customDomains";
import { ftsDeleteProjectStatements } from "../../../api/src/lib/fts";
import { deleteR2Prefix } from "./r2";
import type { Env } from "../index";

Expand Down Expand Up @@ -66,10 +67,11 @@ export async function deleteProjectCascade(
// Docs go explicitly before the project: left to the project's cascade,
// the folders cascade would first SET NULL docs.folder_id on every
// foldered doc (a row update each) only for the docs cascade to delete
// them. docs_fts is a standalone FTS table (no triggers), so its order
// them. docs_fts (+ its docs_fts_map rowid map) is a standalone FTS table
// (no triggers, found by the site's pid token), so its order
// relative to the docs delete doesn't matter.
await env.DB.batch([
env.DB.prepare("DELETE FROM docs_fts WHERE project_id = ?").bind(projectId),
...ftsDeleteProjectStatements(env.DB, projectId),
env.DB.prepare("DELETE FROM docs WHERE project_id = ?").bind(projectId),
env.DB.prepare("DELETE FROM projects WHERE id = ?").bind(projectId),
]);
Expand Down
24 changes: 18 additions & 6 deletions packages/admin/src/routes/projects.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ import { describe, it, expect, vi, beforeEach } from "vitest";
import { Hono } from "hono";

vi.mock("../audit", () => ({ writeAdminAudit: vi.fn() }));
vi.mock("../../../api/src/lib/fts", () => ({ upsertFtsRow: vi.fn() }));
vi.mock("../../../api/src/lib/customDomains", () => ({
releaseCustomDomain: vi.fn(),
removeCustomDomain: vi.fn(),
Expand Down Expand Up @@ -300,13 +299,17 @@ describe("DELETE /:id - scale safety (AB-H1)", () => {
expect(chunks.flatMap(c => c.binds)).toEqual(docIds.map(d => d.id));
expect(calls.some(c => c.sql.includes("doc_shares"))).toBe(false);

// One final batch: FTS rows (map first, then the rows, both via the
// site's pid token), the site's docs, then the project row.
expect(batches.length).toBe(1);
const finalBatch = batches[0];
expect(finalBatch[0].sql).toContain("docs_fts");
expect(finalBatch[1].sql).toContain("DELETE FROM docs WHERE project_id = ?");
expect(finalBatch[1].binds).toEqual(["proj-1"]);
expect(finalBatch[2].sql).toContain("DELETE FROM projects");
expect(finalBatch[0].sql).toContain("DELETE FROM docs_fts_map");
expect(finalBatch[1].sql).toContain("DELETE FROM docs_fts WHERE docs_fts MATCH ?");
expect(finalBatch[1].binds[0]).toMatch(/^pid:"[0-9A-F]+"$/);
expect(finalBatch[2].sql).toContain("DELETE FROM docs WHERE project_id = ?");
expect(finalBatch[2].binds).toEqual(["proj-1"]);
expect(finalBatch[3].sql).toContain("DELETE FROM projects");
expect(finalBatch[3].binds).toEqual(["proj-1"]);

expect(releaseCustomDomain).toHaveBeenCalledWith(expect.anything(), "proj-1");
expect(writeAdminAudit).toHaveBeenCalledTimes(1);
Expand Down Expand Up @@ -382,12 +385,21 @@ describe("POST /:id/reindex (AB-H2)", () => {
});

it("reindexes a small project and audits after success", async () => {
const { db, firstQueue, allQueue } = makeDb();
const dbRef = makeDb();
const { db, firstQueue, allQueue } = dbRef;
firstQueue.push({ id: "proj-1" });
allQueue.push([{ id: "d1", title: "One" }, { id: "d2", title: "Two" }]);
const request = makeApp({ DB: db, ASSETS: r2 });
const res = await request("/proj-1/reindex", { method: "POST" });
expect(res.status).toBe(200);
// One batch for the chunk (3 statements per doc), then the stale-row prune.
const { batches } = dbRef;
expect(batches).toHaveLength(2);
expect(batches[0]).toHaveLength(6);
expect(batches[0][1].sql).toContain("INSERT INTO docs_fts");
expect(batches[0][1].binds).toContain("d1");
expect(batches[0][4].binds).toContain("d2");
expect(batches[1][1].sql).toContain("doc_id NOT IN (SELECT id FROM docs WHERE project_id = ?)");
expect(writeAdminAudit).toHaveBeenCalledWith(
expect.anything(), session, "project.reindex", "project", "proj-1", { indexed: 2 },
);
Expand Down
10 changes: 7 additions & 3 deletions packages/admin/src/routes/projects.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { Hono } from "hono";
import { upsertFtsRow } from "../../../api/src/lib/fts";
import { ftsUpsertStatements, ftsPruneProjectStatements } from "../../../api/src/lib/fts";
import { removeCustomDomain } from "../../../api/src/lib/customDomains";
import { writeAdminAudit } from "../audit";
import { type KeysetCursor, encodeCursor, decodeCursor, keysetClause } from "../lib/cursor";
Expand Down Expand Up @@ -151,18 +151,22 @@ projectsRouter.post("/:id/reindex", async (c) => {
}, 409);
}

// Each doc's FTS row is replaced by rowid; one D1 batch per chunk (3
// statements per doc), then rows of docs no longer in the site are pruned.
let indexed = 0;
for (let i = 0; i < docs.results.length; i += REINDEX_CHUNK) {
const chunk = docs.results.slice(i, i + REINDEX_CHUNK);
await Promise.all(
const stmts = await Promise.all(
chunk.map(async (doc) => {
const obj = await c.env.ASSETS.get(`${projectId}/${doc.id}`);
const content = obj ? await obj.text() : "";
await upsertFtsRow(c.env.DB, doc.id, projectId, doc.title, content);
return ftsUpsertStatements(c.env.DB, doc.id, projectId, doc.title, content);
}),
);
await c.env.DB.batch(stmts.flat());
indexed += chunk.length;
}
await c.env.DB.batch(ftsPruneProjectStatements(c.env.DB, projectId));

await writeAdminAudit(c.env, session, "project.reindex", "project", projectId, { indexed });
return c.json({ ok: true, data: { indexed } });
Expand Down
47 changes: 47 additions & 0 deletions packages/api/migrations/0064_rebuild_docs_fts.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
-- Rebuild docs_fts so per-site search and per-doc writes stop scanning the
-- whole platform's FTS corpus (see packages/api/src/lib/fts.ts).
--
-- 1. `pid` is a new INDEXED column holding one token per row: hex(project_id).
-- Search ANDs `pid:"<token>"` into the MATCH, so FTS5 only evaluates (and
-- runs bm25/snippet for) rows of the requested site. hex() of the UTF-8 id
-- is injective, a single token for any id under the ascii tokenizer, and
-- (72 chars for a UUID, > the porter stemmer's 64-char cap) never stemmed.
-- pid is declared after body, so snippet(docs_fts, 1, ...) is still the body
-- and bm25(docs_fts, 5.0, 1.0, 0.0) keeps the title/body weights while
-- giving the pid hit no weight.
-- 2. docs_fts_map(rid, doc_id) maps each doc to its FTS rowid, so writes are
-- rowid lookups instead of a `WHERE doc_id = ?` scan of an UNINDEXED column.
-- Invariant (kept by lib/fts.ts and lib/trash.ts): exactly one map row per
-- docs_fts row.
--
-- Rows are copied with their rowids preserved. Only one row per doc survives
-- (the newest, should a race have left duplicates), and rows whose doc no
-- longer exists in `docs` are dropped: a binned doc's FTS row already lives in
-- trash_items.extra_json (restore re-inserts it from there), so anything left
-- in docs_fts for a missing doc is a leak (e.g. a site deleted without its
-- FTS cleanup).

CREATE VIRTUAL TABLE docs_fts_new USING fts5(
title,
body,
pid,
doc_id UNINDEXED,
project_id UNINDEXED,
tokenize = 'porter ascii'
);

INSERT INTO docs_fts_new (rowid, title, body, pid, doc_id, project_id)
SELECT f.rowid, f.title, f.body, hex(f.project_id), f.doc_id, f.project_id
FROM docs_fts f
WHERE f.rowid IN (SELECT max(rowid) FROM docs_fts GROUP BY doc_id)
AND f.doc_id IN (SELECT id FROM docs);

DROP TABLE docs_fts;
ALTER TABLE docs_fts_new RENAME TO docs_fts;

CREATE TABLE docs_fts_map (
rid INTEGER PRIMARY KEY,
doc_id TEXT NOT NULL UNIQUE
);

INSERT INTO docs_fts_map (rid, doc_id) SELECT rowid, doc_id FROM docs_fts;
76 changes: 76 additions & 0 deletions packages/api/scripts/repair-docs-fts.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
-- Re-runnable repair for docs_fts / docs_fts_map (migration 0064).
--
-- Run it once right after the api + admin workers that understand 0064 are
-- fully deployed. Between applying 0064 and that deploy, still-running old code
-- writes docs_fts rows with no `pid` and no map row, and deletes rows without
-- touching the map. Search ignores those rows (they can't match `pid:`), so the
-- only symptom is that the affected docs are missing from search until this
-- runs. Safe to run any number of times; on a consistent index it changes
-- nothing.
--
-- cd packages/api && npx wrangler d1 execute cubedocs-main --remote \
-- --command="$(grep -v '^--' scripts/repair-docs-fts.sql | tr '\n' ' ')"
--
-- (Git Bash/POSIX shell.) Why not plain `--file` / `--command "$(cat ...)"`:
-- a remote --file goes through D1's bulk-import API rather than the query
-- path `migrations apply` uses (the path that created and rebuilt docs_fts);
-- and on Windows `npx` runs through a .cmd shim that cuts an argument off at
-- its first newline, so a multi-line --command silently runs only its first
-- line. Stripping the full-line comments and joining the lines avoids both;
-- the output must show 8 result sets, the last two being the checks below.
-- Every step is defined on the current state alone, so a run interrupted
-- part-way is fixed by simply running it again.
--
-- Old code always deletes EVERY row of a doc before inserting its new one, so
-- if a doc has both a mapped row and unmapped rows, the mapped row was written
-- later (by new code) and is the one to keep.

-- 1. Rows whose doc no longer exists (binned docs keep theirs in
-- trash_items.extra_json; anything else is a leak).
DELETE FROM docs_fts WHERE rowid IN (
SELECT rowid FROM docs_fts WHERE doc_id NOT IN (SELECT id FROM docs)
);

-- 2. Map rows that don't point at their own doc's row.
DELETE FROM docs_fts_map WHERE NOT EXISTS (
SELECT 1 FROM docs_fts f WHERE f.rowid = docs_fts_map.rid AND f.doc_id = docs_fts_map.doc_id
);

-- 3. Unmapped rows of a doc that still has a mapped row: stale.
DELETE FROM docs_fts WHERE rowid IN (
SELECT f.rowid FROM docs_fts f
WHERE f.rowid NOT IN (SELECT rid FROM docs_fts_map)
AND f.doc_id IN (SELECT doc_id FROM docs_fts_map)
);

-- 4. Remaining unmapped rows: keep only the newest per doc.
DELETE FROM docs_fts WHERE rowid IN (
SELECT rowid FROM docs_fts
WHERE rowid NOT IN (SELECT rid FROM docs_fts_map)
AND rowid NOT IN (
SELECT max(rowid) FROM docs_fts WHERE rowid NOT IN (SELECT rid FROM docs_fts_map) GROUP BY doc_id
)
);

-- 5. Map them.
INSERT INTO docs_fts_map (rid, doc_id)
SELECT rowid, doc_id FROM docs_fts WHERE rowid NOT IN (SELECT rid FROM docs_fts_map);

-- 6. Give rows written by old code their site token.
UPDATE docs_fts SET pid = hex(project_id) WHERE pid IS NULL OR pid <> hex(project_id);

-- Verification: fts_rows = map_rows, bad_pid = 0. unindexed_docs counts docs
-- with no FTS row at all (normally 0; e.g. a doc binned by new code while its
-- row was still an unmapped old-code row, then restored). If it isn't 0,
-- reindex those sites (admin Sites -> Reindex, or the owner's
-- POST /projects/:id/reindex) - the per-site breakdown is below.
SELECT
(SELECT count(*) FROM docs_fts) AS fts_rows,
(SELECT count(*) FROM docs_fts_map) AS map_rows,
(SELECT count(*) FROM docs_fts WHERE pid IS NULL OR pid <> hex(project_id)) AS bad_pid,
(SELECT count(*) FROM docs d WHERE NOT EXISTS (SELECT 1 FROM docs_fts_map m WHERE m.doc_id = d.id)) AS unindexed_docs;

SELECT d.project_id, count(*) AS unindexed_docs
FROM docs d
WHERE NOT EXISTS (SELECT 1 FROM docs_fts_map m WHERE m.doc_id = d.id)
GROUP BY d.project_id;
Loading
Loading