Repository navigation
feat(worker): retain raw agent output in R2 with a per-attempt cap - #142
Conversation
Store each output chunk as an R2 object at output/<job>/<attempt>/<seq>, indexed in D1 by a reservation row with the text's hash. Chunks must arrive with a contiguous seq; identical retries succeed without rewriting, and a conflicting chunk is refused before it reaches R2. Each attempt keeps at most 64 MiB and 4096 chunks; past either cap chunks get a `truncated` reply and the job keeps running. A bucket lifecycle rule expires output after 30 days and a daily cron deletes the index rows. Disconnecting a project queues its jobs' output for deletion in the same D1 batch, deletes the prefixes at once, and the cron retries failures. An upload racing a disconnect deletes its own object. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02456795c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(quest): add R2 output bucket setup to the kix.quest deploy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(quest): make the deploy's R2 step depend on #142 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A pending chunk could fill a page and advance the cursor past it, so a reader following `next` missed it once a retry completed it. Reads now list only stored chunks, a chunk is reserved only after the one before it is stored, and a pending chunk completes only while its attempt is current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b0975be38b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… self-cleanup A chunk whose first upload failed kept its reservation time, so the index could expire before the object the lifecycle rule had just restarted. Completing a reservation now records the upload time. An upload that races a disconnect queues its job's prefix for cleanup again before deleting its own object, so a failed delete is retried by the cron. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # quest/m0/kix/README.md
|
Merge summary Raw agent output now lives in R2 ( Review: Codex raised three P2 findings over two rounds, all accepted and fixed (b0975be: pagination never passes a chunk that may still land; ef024f4: retention starts at upload time, and a failed self-delete re-queues cleanup). Codex then completed code and security reviews on 7a179e8 with no new findings. The three addressed threads are resolved. Two of the three automated fix rounds were used. CI is green on 7a179e8, and the branch is current with main. #144 and #145 also add migration 0008 and have not merged, so this PR keeps Enabling auto-merge on 7a179e8. (Written by Claude Opus 5.5) |
Brings in raw output retention (#142) and the harness and answer-flow plans (#147). The inbox migration becomes 0009_inbox.sql after 0008_output.sql. The shared maintainerOf() fixture adopts main's stricter GitHub mock, where only the signed-in user holds the maintain role. harness.md and answer-flow.md drop this deleted quest from Required, and activity.md drops both deleted quests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Completes
quest/m0/kix/output.md(Retain raw agent output in R2).Problem
#135 stored raw agent output as text in D1
job_outputrows, with no total cap and no retention. Launch would ship unbounded transcripts in D1.Behavior
output/<job>/<attempt>/<seq>in thequest-outputbucket (bindingOUTPUT). D1 keeps one small index row per chunk (hash, complete flag, upload time) and one totals row per attempt (job_output_attempts). Migration0008_output.sqldrops the old table; nothing is deployed, so there is no data copy.seq. An attempt accepts only its nextseq, starting at 0, or an identical retry of a chunk still indexed. Each chunk is reserved first: its index row and hash are inserted in a D1 batch, only while the attempt is current. A conflicting chunk is refused before anything reaches R2. A chunk is reserved only once the one before it is stored, so an attempt's one pending chunk is its last. Then the object is written and the row is marked complete with the upload time, only while the attempt is still current. A retry that matches a completed row writes nothing, so the object's lifecycle never restarts. A replay of an expiredseqis refused.truncatedonce, and it and every later chunk get{ "type": "reply", "id": …, "truncated": true }instead of an error. The job and its summaries are unaffected. Expiry never gives capacity back, because the totals row lives as long as the job.GET …/jobs/<job>/outputstill pages by(attempt, seq)behind the per-request maintainer check. It lists only stored chunks, so a cursor never passes a chunk that may still land, skips rows whose object is already gone, and addstruncated: [attempts].output/after 30 days (documented, since lifecycle is bucket config). A daily cron (17 4 * * *) deletes index rows older than 30 days and retries pending cleanups. The Worker's default export is now{ fetch, scheduled }, and the Hono app is the named exportapp, which the tests import.output_cleanuprow per job with output. The route then deletes eachoutput/<job>/prefix and its row, and the cron retries any failures. An upload that races the disconnect finds its row gone after writing, queues its job's cleanup again, and deletes its own object.cloudflare/README.mdgets a new "Raw output" section and the bucket and lifecycle deployment commands. The runner protocol note inquest/m0/kix/runner.mdis updated. The finished quest is deleted and its references removed.Validation
just checkandjust testpass innix develop(225 CLI tests, 212 Worker tests). New Worker tests against local R2 cover:scheduledhandler, including refused replays and skipped missing objectsThe route test also checks that signed-out visitors, other signed-in GitHub users, and revoked maintainers are refused.
Settled decisions
The maintainer accepted each recommendation:
MAX_CHUNKSstays 4096 (cloudflare/src/jobs/output.ts). Reaching 64 MiB needs chunks of at least 16 KiB on average, and an attempt costs at most about 8k R2 Class A operations.truncated: true, not anerror, so a runner never mistakes it for a failure to retry.job_output_attemptsrows live as long as the job and are deleted with it.Suggested follow-ups
r2 bucket create quest-outputand the 30-day lifecycle rule in the deploy checklist.jobs.actor) leave their R2 objects until the lifecycle rule expires them. Route any future job-deletion path throughoutput_cleanup.🤖 Generated with Claude Code
(Written by Claude Opus 5.5)