Repository navigation
Take jupyter-book out of the paper render environment and uv.lock - #227
Conversation
The docs extra was both the Quarto paper render environment and the Jupyter Book 2 docs-site builder. Quarto's Jupyter engine needs only the kernel and notebook runner (ipykernel, nbclient, nbformat); jupyter-book pulled in jupyter-server, nbconvert, bleach, mistune, soupsieve and 29 other packages that the render never imports. Drop jupyter-book from the docs extra and let docs/requirements.txt, which already declared it, be its only declaration. A separate extra would not help: Dependabot scans every package in uv.lock whichever extra pulls it in. The relock removes 34 packages and changes no version. The paper re-renders identically (HTML text and PDF bytes, apart from Quarto's per-render cell id and the PDF dates and /ID), and the docs site builds through docs/requirements.txt with the same page text as main's locked jupyter-book. README and paper/README document both environments; tests/test_render_environment.py keeps the stack out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Taking this PR over from session local_a36e319c, which has been idle since 2026-10-14; I've messaged it. Review r1 (Sol) approved at dd09f15 with one nit. This rebases on #225 once it merges (main's lock, then |
Review r1's nit: an indented comment would have reached _name() and failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uv.lock conflicted with #225's bumps. Took main's lock and ran `uv lock` (uv 0.13.4): the same 34 packages leave, none is added and no retained version changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (review r2) The guard test's reader now joins backslash continuations and strips comments as pip's req_file.py does, follows -r includes relative to the including file, skips -c constraints and other options, and drops per-requirement options such as --hash. Requirements it can't name without fetching them (bare URLs, paths, -e) fail the test instead of yielding a wrong name. Names come from packaging's PEP 508 parser. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (review r3) The reader of docs/requirements.txt tried to follow pip, and review r3 found four families of lines on which the test would pass while pip installs no jupyter-book: unevaluated markers, PEP 508 strings pip reads as archive paths (==2.0+local.zip), -r spellings pip reads differently (-r=FILE, -rFILE), and a ";" inside a named URL. uv, which builds the docs site, differs from pip on some of these too: it rejects a backslash continuation inside a requirement and reads -r=FILE as FILE. The reader now accepts only blank lines, comments, -r FILE / --requirement FILE / --requirement=FILE includes, and requirements made of a name, extras and version specifiers. Every other line fails with a message naming why: continuations, whitespace other than spaces and tabs (the file is read as bytes, so a lone CR isn't a line break), other options, per-requirement options, URLs and paths, markers, ===, and archive names as pip judges them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ew r4) Review r4 found that pip decodes a requirements file by a PEP 263 line among its first two (_decode_req_file in req_file.py), so an accepted comment such as "# coding: utf-7" could make pip join a later line into a comment and install no jupyter-book; uv ignores the line. The reader now refuses an encoding declaration in a file's first two lines, a UTF-8 byte-order mark, and bytes that aren't UTF-8. It also refuses: - an include cycle, which pip reports as an error and uv reads as nothing; - an include whose .. comes after a symlinked directory, which pip collapses lexically while uv follows the link, so they read different files; - a trailing comma in the version specifiers, which uv 0.11.7 rejects and pip and uv 0.13.4 accept. And it strips extras together with any whitespace before them before the archive-name check, as pip's strip_extras does, so "jupyter-book.zip [docs]" is refused. A correction to c5744ce's message: it said uv rejects a backslash continuation inside a requirement, with no logged run behind it. Review r5's differential now records one: uv 0.13.4 can't parse "jupyter-book \" followed by ">=2.0", which pip joins and installs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r5 approved 919b988 and listed forms the differential had never run. Run now: pip installs "jupyter-book ()" and "-r base.txt/" (or "base.txt/."), which pathlib also reads as base.txt, while uv 0.11.7, 0.12.13 and 0.13.4 all reject them. The reader now refuses both, so it again accepts only forms the two read alike. An include inside a file reached through a symlink, an absolute include path, a tab after --requirement, and pre, post and dev releases are read alike and stay accepted; the reader test now covers the absolute path and the tab. A correction to 919b988's message: it said uv reads an include cycle as nothing. That holds for a file whose only line includes itself; with "foo" before the self-include, uv installs foo while pip reports the cycle. The reader refuses both, and both are now among the refused lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r6 approved d3a430e and noted that the reader test's absolute include depended on the temp directory's path matching the include pattern, so a temp root or user name with a space, @ or + would fail the test falsely. It also asked whether uv reads a symlinked file reached under two names as pip does. Run now, pip and uv 0.11.7, 0.12.13 and 0.13.4 read an absolute include, an include of a symlinked file, and that file reached both directly and through the link alike. The reader refuses both forms anyway: no file here needs them, and refusing them keeps the test independent of the machine's paths and of how uv recognizes a file it has already read. Also from the review: the comment on paths ending in / now says uv rejects them, which is what the log shows, rather than why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view r7) Review r7 approved a381226 with two non-blocking findings in the reader. The symlink check looked only at the path as the OS resolves it. With "-r sub/../x.txt", where docs/sub links to a directory and docs/x.txt is a symlink, pip opens docs/x.txt (it collapses .. first) while the OS follows sub and finds a regular file, so the include was accepted. pip and uv 0.11.7, 0.12.13 and 0.13.4 read that tree alike, so no name was wrong, but the comment said "names no symlink". The check now looks at both paths, and the reader test builds that tree. An absolute include was refused only as "an option", and its one test line named a file that doesn't exist, so the mutant allowing absolute paths died of a missing file rather than an accepted include. Absolute includes now have their own message, and a new test includes a real file by its absolute path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merge audit. Squash-merged at head Gates at that head:
Independent reviews, saved verbatim in
Round 8 approved the merged head. Its non-blocking notes are answered in the body. 🤖 Generated with Claude Code |
The
docsextra did two jobs: it was the environment for the Quarto paper render, and it held the Jupyter Book 2 builder for the operational-docs site indocs/. The render needs only a Jupyter kernel and notebook runner. jupyter-book brought jupyter-server, nbconvert, bleach, mistune, soupsieve and 29 more packages intouv.lock, and the render uses none of them. This PR takes jupyter-book out ofpyproject.toml.docs/requirements.txtalready declared it and is now its only declaration. The relock removes 34 packages and changes no version.Why not a separate
docs-siteextraDependabot scans every package in
uv.lock, whichever extra pulls it in. bleach, mistune, soupsieve and jupyter-server are reachable only through thedocsextra (uv tree --invert:jupyter-book→jupyter-server→nbconvert). Yet all four had open alerts againstuv.lockin the 2026-10-14 inventory (alerts_before.tsv), listed as transitive runtime dependencies, until #225's bumps closed them. Adocs-siteextra would keep them in the lock, and their future alerts with them. Taking jupyter-book out of the lock is what removes them. GitHub's dependency graph recordsdocs/requirements.txtonly asjupyter-book >= 2.0, with no resolved version and no transitive packages.Dependabot alerts
#225 (merged as
555a0f2c) bumped bleach, jupyter-server, mistune and soupsieve to patched versions, and their 18 alerts are now fixed: bleach #42–#44, jupyter-server #45, mistune #49–#58, soupsieve #47, #48, #69 and #70. Bleach #42 (GHSA-g75f-g53v-794x) closed only because the advisory's range is= 6.3.0. 6.4.0 keeps the same email regex, and no release fixes it. This PR removes all four packages fromuv.lock, so that code is no longer installed at all.Five alerts are open on main after #225:
tornado's alerts (#66–#68, #77–#79) were fixed by #225, and tornado stays in the lock because ipykernel and jupyter-client need it.
What leaves
uv.lockField-by-field comparison with main's lock (
555a0f2c, after #225): 171 → 137 packages; 34 removed, 0 added, 0 version changes. Relocking on #225's lock removes the same 34 packages as relocking on the lock before it.argon2-cffi 25.1.0, argon2-cffi-bindings 25.1.0, arrow 1.4.0, beautifulsoup4 4.14.3, bleach 6.4.0, defusedxml 0.7.1, fqdn 1.5.1, isoduration 20.11.0, jsonpointer 3.0.0, jupyter-book 2.1.2, jupyter-events 0.12.0, jupyter-server 2.21.1, jupyter-server-terminals 0.5.4, jupyterlab-pygments 0.3.0, lark 1.3.1, mistune 3.3.4, nbconvert 7.17.1, nodeenv 1.10.0, overrides 7.7.0, pandocfilters 1.5.1, prometheus-client 0.24.1, python-json-logger 4.0.0, pywinpty 3.0.3, rfc3339-validator 0.1.4, rfc3986-validator 0.1.1, rfc3987-syntax 1.1.0, send2trash 2.1.0, soupsieve 2.10, terminado 0.18.1, tinycss2 1.4.0, uri-template 1.3.0, webcolors 25.10.0, webencodings 0.5.1, websocket-client 1.9.0.
Two retained entries change in place:
policybench: itsdocsextra loses jupyter-book.jsonschema4.26.0: it loses itsformat-nongplextra entry, which only jupyter-events requested.The lock's top-level metadata is unchanged.
Why the render doesn't need jupyter-book
Quarto 1.9.36's Jupyter engine (
share/jupyter/notebook.py,jupyter.py) importsjupyter_client,nbclient,nbformatandyaml. Its two other imports, papermill (for--execute-params) and jupyter-cache (forcache: true), sit intry/except ImportError, and neither was ever locked. The kernel is ipykernel'spython3. All of these remain in thedocsextra or come in with it. In the synced venv,jupyter_server,nbconvert,bleach,mistune,soupsieve,bs4,jupyter_bookandterminadoare all absent, andquarto check jupyterpasses.Changes
pyproject.toml:docs = ["ipykernel>=6.0", "nbformat>=5.0", "nbclient>=0.10"], with a comment saying what the extra is for and where jupyter-book went.uv.lock: relocked withuv lock(uv 0.13.4) on main's lock after Patch 50 of 53 Dependabot alerts in uv.lock; document diskcache and bleach, which have no fix #225. Only the removals and the two in-place entries above change.docs/requirements.txt: unchanged requirement (jupyter-book>=2.0), plus a comment with the build command.README.md: a new section, "Paper and documentation site", documents both environments and the docs build command.paper/README.md: says what thedocsextra holds and that it excludes jupyter-book.tests/test_render_environment.py: guard tests.The
docsextra holds ipykernel, nbclient and nbformat.No extra declares jupyter-book, jupyter-server or nbconvert.
None of those three is in
uv.lock.docs/requirements.txtdeclares jupyter-book. The test's reader accepts only line forms that pip and uv (which builds the docs site) read the same way. It fails on every other line rather than guessing (reviews r3 to r7). The accepted forms:#at a line's start or after whitespace), except a PEP 263 encoding line among a file's first two;-r FILE,--requirement FILEor--requirement=FILE, where FILE is a relative path, resolved against the including file's directory, that names no symlink (checked both with..collapsed, as pip opens it, and as the OS resolves it), has no..after a symlinked directory, doesn't end in/or/., and closes no cycle;Every file must be UTF-8 with no byte-order mark. Each refusal has its own message, and
r5/differential.logrecords what pip and uv do with each refused line:jupyter-book \onto a following>=2.0and installs it, while uv can't parse the pair. And a comment that doesn't start its line but ends in\makes pip swallow the next line:foo # see below \followed byjupyter-bookinstalls only foo with pip, and both with uv.str.splitlines. A form feed inside a requirement makes pip fail while uv installs it, and a no-break space before a comment is a parse error for uv while pip installs. The file is read as bytes, so a lone carriage return is caught too.coding[:=]comment among its first two lines (_decode_req_fileinreq_file.py); uv ignores such a comment. So# coding: utf-7followed byfoo # x+AFwandjupyter-bookinstalls only foo with pip (in UTF-7,+AFwis a backslash), and both with uv.# Docs site; no coding=neededmakes pip fail on the unknown codecneeded. A UTF-8 byte-order mark is read the same way by both, but it is refused anyway, since only the plain UTF-8 path is modelled.foobefore a self-include installs foo, and a two-file cycle installs both files' requirements...comes after a symlinked directory, which pip collapses lexically while uv follows the link, so the two read different files./or/., which pip (like pathlib) reads as the file and uv rejects.sub/.., while the OS, followingsub, reaches a regular file. They are refused anyway: no file here needs them, and refusing them keeps the test independent of the machine's paths and of how uv recognizes a file it has already read.-r=FILE(pip opens=FILE, uv opensFILE);-rFILEand--requirem FILE(pip follows them, uv rejects them);$-expanded or URL paths, and includes with trailing options;-cand-ein any spelling (--pre -e ./localmakes pip fail and uv install nothing);--hash.===, string equality, which the reader doesn't model. Both tools read===2.0alike, but===2.0.zipis a file path to pip and a parse error to uv.jupyter-book ()), which pip accepts and uv rejects.utils/filetypes.py). Sojupyter-book==2.0+local.zipis refused: pip reads it as a file path, uv as a version.Five more tests pin that reader:
..through it is refused;sub/..;A differential against real pip and three uv versions, and 30 mutants, are under Verification.
Run against main's pyproject and lock, the two lock and extra checks fail as intended.
Verification
Local runs used CPython 3.14.7 (GIL build) on macOS arm64. Evidence is in
~/reviews/policybench-docs-site-extra-2026-10-14/(r3/for the render and lock,r5/for the requirements reader).Render environment:
uv sync --locked --extra dev --extra docs(uv 0.13.4) succeeds, anduv pip checkfinds the 134 installed packages compatible.jupyter_server,nbconvert,bleach,mistune,soupsieve,bs4,jupyter_bookandterminadodon't import.Paper, with and without the 34 packages. The same source was rendered twice: main at
555a0f2c, and this branch atb6f6f778, which changes onlypaper/README.mdunderpaper/,policybench/andapp/(render_paper.pyrendersindex.qmdalone). It was rendered withpaper/render_paper.py(HTML and PDF, Quarto 1.9.36, LuaHBTeX), main in its full locked environment in a separate worktree, this branch in its own. Both exit 0.web/index.htmldiffers on one line, a Quarto per-render cell id (id="965529f8"→id="e2561682"). Its visible text is identical (0 diff lines).policybench.pdfhas identicalpdftotext -layouttext and the same size (261,164 bytes). Its 66 differing bytes all fall inside/CreationDate,/ModDateand the trailer/ID; with those masked, the files are byte-identical.Paper at the merged head (
d9f862d1, which brings in Disclose why 330 frozen households record near-zero latency and two models record none #228's paper changes): rendered in this branch's environment and compared with the render Disclose why 330 frozen households record near-zero latency and two models record none #228 committed. 17 of 19 files are byte-identical.web/index.htmldiffers on two lines, both per-render cell ids, with identical visible text.policybench.pdfhas identical text and size (263,255 bytes), and its 70 differing bytes all fall inside/CreationDate,/ModDateand/ID. I restored the committed render, so therendered_paper_artifactspins are untouched.Docs site (round 1, at
dd09f15e;docs/requirements.txthasn't changed since):cd docs && uv run --no-project --with-requirements requirements.txt jupyter book build --htmlexits 0 with jupyter-book 2.1.7 and builds the 4 pages. For comparison I built the same site in main's lockeddocsenvironment (uv export --frozen --extra docsfrom main's lock: jupyter-book 2.1.2).#inside$…$inbenchmark_card.md.keyids.Lock:
uv lock --check(uv 0.13.4) passes atd9f862d1.uv.lockandpyproject.tomlare unchanged by the merge of Disclose why 330 frozen households record near-zero latency and two models record none #228.Lint: at
37c7bfbc,ruff check .passes, andruff format --check .reports 181 files already formatted.uv lock --check(uv 0.13.4) passed at31571187; nothing since touchesuv.lockorpyproject.toml.Tests:
OPENBLAS_NUM_THREADS=1 pytest -m "not slow" -q, run serially atb6f6f778, gives 5459 passed, 58 skipped, 36 deselected (r3/pytest-not-slow.log). At37c7bfbc,tests/test_render_environment.pypasses (65 tests); CI runs the not-slow suite there.The requirements reader against real pip and uv (
r5/differential.py,differential.log, at37c7bfbc; the log starts with the command line and tool versions). Hypothesis (derandomized) generated 300 requirements trees in the accepted forms.docs,docs/subandother../andsub/..forms and go up and across directories. The entry file includes another file in 138 trees, and includes nest up to four deep (in 11 trees).[]and[ docs , x ]), specifiers from a grammar (parentheses, tabs, wildcards, local and epoch versions), comments, CRLF endings and missing final newlines.Every tool ran from an empty directory outside the tree, given the entry file's absolute path. On all 300 trees, the reader's names equal four things:
pip install --dry-run --report, from stub wheels without dependencies);uv pip compile);uv run --no-project --with-requirements.The log prints how often each feature occurred, what the generator filtered out (lines packaging rejects or no stub version satisfies), and one example tree per nesting depth.
It then checks 20 fixed accepted rows on every tool, where each must return the reader's names:
>=v2.0,==2.0-1and==2.0_post1;./and tab includes;docs/sub), not againstdocs/, where a decoy sits beside the link;docs/requirements.txt.Finally it records what pip and each uv do with 64 refused inputs: the 57 refused lines; a
..after a symlinked directory; an absolute include of a real file; an include of a symlinked file; that file reached both directly and through the link, in both orders; a symlinked file that pip opens once it collapsessub/.., while the OS, followingsub, reaches a regular file; and a two-file cycle.r5/pip-errors.txthas pip's last error line for three of them.Mutants (
r5/mutants.py,mutants.log, run on the committed head37c7bfbcwith a clean tree throughr5/run_mutants.sh, which writes the log's first line; every test runs, and each mutant's failing tests are listed under it): 30 of 30 are killed. Each removes or loosens one rule:CI:
d9f862d1: CI run 40176708922 (lint, app, test) and Paper run 40176708957 pass.60f76fa8: CI run 40178391332 and Paper run 40178391363 pass.31571187: CI run 40181871961 and Paper run 40181871759 pass.919b9887: CI run 40184706164 and Paper run 40184706155 pass.d3a430ec: CI run 40187540436 and Paper run 40187540420 pass.a381226a: CI run 40189939694 and Paper run 40189939603 pass.37c7bfbc: CI run 40192194064 and Paper run 40192194090 pass.The Paper workflow re-renders HTML from a clean
uv sync --locked --extra dev --extra docson Python 3.12.Reviews
Round 1 (GPT-6.1 Sol, at
dd09f15e): APPROVE, with one nit: the guard test's requirements reader would fail on an indented comment. Fixed in269d79f3, which handles comments and options as pip does and adds a test of the parser.Round 2 (GPT-6.1 Sol, at
d9f862d1): APPROVE, with three non-blocking corrections, made in60f76fa8and this body:-rincludes, and named bare URLs wrongly.60f76fa8modelled more of pip'sreq_file.py. Round 3 showed that this still missed forms pip names differently, so the reader is now narrowed (round 3, below).Round 3 (Opus 5.5 in-session, at
60f76fa8, the reader only). The Sol run was cancelled before launch because every Subfleet lane was on hold. Verdict: REQUEST_CHANGES.-rspellings pip reads differently, and a;inside a named URL.constructors.py's job, notreq_file.py's.The current one-line file was never affected.
c5744cebtakes the review's second route: it narrows the reader to forms pip and uv agree on and refuses the rest in code.31571187merges main (Record whether each prediction row's responses were live calls or cache replays #231).Round 4 (Opus 5.5 in-session, at
31571187, the reader only; lanes still on hold). Verdict: REQUEST_CHANGES.# coding: utf-7could make pip joinjupyter-bookinto a comment while the test passed.919b9887fixes all of them. It refuses encoding declarations, a UTF-8 byte-order mark and non-UTF-8 bytes, include cycles, includes whose..follows a symlinked directory, and trailing commas, which the widened differential found uv 0.11.7 rejecting. The archive check now strips whitespace before extras. Every input the reviews said should be refused is now among the refused lines, some with different file names. The differential above was widened as asked. Its log now backs the continuation claim, whichc5744ceb's message had made without one;919b9887's message corrects that.Round 5 (Opus 5.5 in-session, at
919b9887, the reader only). Verdict: APPROVE, with nothing blocking. It found no accepted line or tree on which pip or any of the three uv versions quietly installs something else, and checked the reader's packaging 26.0 against pip's vendored 26.2 on 135,456 fuzzed lines with identical results. Non-blocking:-r base.txt/and-r base.txt/.are accepted, but uv probably opens them as written;d3a430ecanswers it. Running the unrun forms showed pip installing-r base.txt/,--requirement=base.txt/.andjupyter-book ()while all three uv versions reject them, so the reader now refuses all three. Includes through a symlinked file, absolute include paths,--requirement<TAB>FILE, pre, post and dev releases,>=v2.0,==2.0-1, names with digits, and non-ASCII and NUL bytes in comments are read alike, so they stay accepted and are fixed rows in the differential (a381226alater refuses the symlinked-file and absolute forms; see round 6). The generator now draws an include from the entry file in about half the trees and logs the nesting depth and each include spelling. The wording is fixed, andd3a430ec's message corrects919b9887's cycle claim.Round 6 (Opus 5.5 in-session, at
d3a430ec, the delta since round 5). Verdict: APPROVE, with nothing blocking. It found every round-5 finding answered with evidence, and the two new refusals reached for the right reason and no broader than the log supports. Non-blocking:@or+there would fail the test falsely;a381226aanswers it. The new rows show pip and uv reading the symlinked file alike in both include orders, an absolute include alike, and a two-file cycle as described above. The reader now refuses absolute include paths and included symlinks anyway, which removes the temp-path dependence. The claims and descriptions are reworded to what the log shows. The pre-release row is==2.1rc1. The log prints one example tree per nesting depth. The mutants were rerun on the committed head.Round 7 (Opus 5.5 in-session, at
a381226a, the delta since round 6). Verdict: APPROVE, with nothing blocking. The delta only narrows what the reader accepts. Non-blocking:-r sub/../x.txt, wheredocs/sublinks to a directory anddocs/x.txtis a symlink, pip opens the symlink, while the OS reaches a regular file. The names still agreed, but "names no symlink" overstated the check.37c7bfbcanswers it. The symlink check now tests both paths, and the reader test builds that tree. Absolute includes have their own message, and a new test includes a real file by its absolute path, which the absolute-path mutant now fails (see round 8). The differential has rows for both trees: every tool reads them alike. The wording is fixed. The mutants ran on the committed head throughr5/run_mutants.sh, which writes the log's first line.Round 8 (Opus 5.5 in-session, at
37c7bfbc, the delta since round 7). Verdict: APPROVE, with nothing blocking. The delta only narrows what the reader accepts. In the reviewer's in-memory fuzz of 60,000 trees with directory and file symlinks, the trees37c7bfbcaccepts are a subset of thosea381226aaccepts, and on none of them do its models of pip and uv disagree. Non-blocking:r5/mutants.pynow runs every test and lists each mutant's failures. Under "absolute include paths allowed",mutants.loglists the new test among three failures, andr5/mutant-reason-absolute.logshows why: it fails with "DID NOT RAISE", because the mutant reads the absolute include instead of refusing it.37c7bfbcwas running; it has passed.docs/sub.sub/.." read as if pip followssub; it now says pip collapsessub/..first.Invariants
This PR changes no logic. It must preserve three properties:
tests/test_render_environment.pyenforces this.axiom: n/a: dependency/environment change, no policy logic🤖 Generated with Claude Code