Repository navigation
Review follow-ups: redact bootstrap source from traces, checkable claims, stricter config, least-privilege release - #26
Merged
Conversation
Since #22 the rich bootstrap goes through gemini.ts's traced client, so with LANGFUSE_* set its prompts (up to 16 MB of the team's source files) and its answers (summaries that can quote that code) were copied into Langfuse. tracedGenerate takes `redact`, and generateText (the bootstrap's entry) sets it: those generations record only the prompt and answer sizes, the model settings and token usage. Request-path calls (score, coach, improve, diff) are traced as before. Two tests with a fake trace: a bootstrap call's source line never reaches the trace (fails with redact off), and a /score prompt still does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The held-out set and the rule-based scorer were added in the same commit (e3c7e2c), so "written before the rules were frozen, never tuned on" can't be verified from history. README, eval README, holdout.json and baseline.test.ts now call it author-attested and say why. - "The floor the Gemini scorer has to beat" (Try-it section, README, eval README, the scorer's header comment) implied a comparison that hasn't been run. Now: a baseline a model scorer should beat; not compared yet. The landing page's copy of the scorer is re-synced (the sync test checks it). - "Watch the 3-min demo" / "3-minute walkthrough": the video's length could not be checked from here (YouTube returned no metadata), so the number is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`TRAILHEAD_DEMO_TEAM=0` or `=no` silently meant "default", which on a server without an admin token is "on" — the opposite of what was probably meant. demo-team.ts now owns the parsing: on/true, off/false, or unset for the default; index.ts exits at startup with a message for anything else, and if an unknown value appears anyway (env changed under a running process) the demo team fails closed. Unit tests for both; checked that `TRAILHEAD_DEMO_TEAM=no` stops `src/index.ts` before it touches the database. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hed releases
- permissions: {} at the top; the build job (checkout, npm ci, tests,
packaging) runs with contents: read, and only the release job, which runs
no repository code (it downloads the build artifact and calls gh), gets
contents: write.
- actions/checkout with persist-credentials: false.
- @vscode/vsce 3.9.2 is an exact devDependency of apps/vscode-ext instead of
an `npx --yes` fetch at release time (lockfile written with npm 11 so the
existing libc fields stay; `npm ci` with Node 22's npm 10.9 accepts it;
npm audit --audit-level=high clean).
- Assets are only uploaded to a draft: if the tag's release is already
published the job fails instead of --clobbering files people may have
downloaded; a draft (e.g. a re-run) can still be refreshed.
Checked: actionlint clean on release.yml and ci.yml; the attach step run
against a stubbed gh creates+uploads with no release, uploads to a draft,
and exits 1 for a published release; `npm run package` builds the .vsix
with the local vsce.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#23 relaxed GEMINI_API_KEY from `:?` to `:-` so `TRAILHEAD_LLM=offline` could run without a key; with neither set, the API then restart-looped with its error in the logs. Compose can't require a variable only when another is unset (nested `${A:-${B:?}}` is evaluated eagerly — tried), so a one-shot `config-check` service (the Postgres image's shell, nothing new to pull) checks it and prints what to set; the API depends on it with service_completed_successfully. Checked with docker compose 2.35: neither set → `up` exits 1 naming the fix and the API never starts; TRAILHEAD_LLM=offline and a key each → `up --wait` exits 0 with the API healthy (so CI's smoke test is unaffected). SELFHOSTING.md: the new failure mode and troubleshooting entry; what reaches Langfuse (rich-bootstrap traces carry sizes and usage only); unknown TRAILHEAD_DEMO_TEAM values stop the API; "floor" reworded to "baseline". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the review of #21–#25. One commit per point.
Why and what changed
ecb096a). Since API: split app.ts into route modules, one Gemini client, remove dead code #22 the rich bootstrap uses the traced client, so its prompts (up to 16 MB of source) and answers were copied into Langfuse traces. Now those generations record only sizes, model settings and token usage. Request-path calls are traced as before. SELFHOSTING says what reaches Langfuse.282493a).holdout.jsonandheuristic-score.mjslanded in the same commit (e3c7e2c). README, the eval README,holdout.jsonandbaseline.test.tsnow call the separation author-attested and say why.282493a). The Gemini eval hasn't been run. The Try-it section, README, eval README, SELFHOSTING and the scorer's header comment now say "a baseline a model scorer should beat; not compared yet". The landing-page copy of the scorer is re-synced.282493a). I couldn't verify the length: YouTube returned no metadata to the page fetch or the player API. The number is dropped from the hero and the README.TRAILHEAD_DEMO_TEAMvalues (fc4b713). Values like0ornoused to mean "default". The API now exits at startup naming the allowed values, and fails closed if one appears at runtime.release.yml(13eb3f3).permissions: {}at the top.contents: read. Only the release job, which runs no repo code, hascontents: write.persist-credentials: falseon checkout.@vscode/vsce3.9.2 pinned as a devDependency instead of fetched withnpx.--clobbering its files.12af697). Nested interpolation is evaluated eagerly (tried), so a one-shotconfig-checkservice (the Postgres image's shell) stopsdocker compose upwith a message when there is neither a key norTRAILHEAD_LLM=offline. The API depends on it viaservice_completed_successfully.Verification
npm run typecheck/npm run lint/npm audit --audit-level=highnpm testTRAILHEAD_DEMO_TEAM=no npx tsx src/index.tsactionlint(1.7.12) on both workflowsghnpm ciwith npm 10.9 (Node 22) on the new lockfilelibcfields are keptnpm run package(local vsce).vsixdocker compose2.35, neither key nor offlineupexits 1 with the message; the API never startsdocker compose2.35, offline / with a keyup -d --waitexits 0 with the API healthy, so CI's smoke test is unaffected🤖 Generated with Claude Code