[MaxGhenis]

PolicyBench had 44 open Dependabot alerts when this PR opened on 2026-10-14, all on Python packages pinned in uv.lock. This PR upgrades the locked versions of the 15 packages behind them to patched releases. bleach's GHSA-g75f should also close, but only because its advisory names a single version; the bug is not fixed (see below).

Nine more alerts opened while this PR waited, #84–#92 (count on 2026-10-25: 53 open, the 44 plus these nine):

• aiohttp Make Claude runs parallel-safe by moving the hard timeout to the chunk subprocess #84–Triage 16 reference-suspect cases from the Claude 5 failure audit #91 (three high) are all fixed in 3.14.4. This PR's aiohttp 3.14.5 covers them, so they close with the rest.
• nbconvert scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92 (high, GHSA-pjvx-prjx-h659, fixed in 7.17.2) is not bumped here. nbconvert is in the lock only through jupyter-book, and Take jupyter-book out of the paper render environment and uv.lock #227, which merges right after this PR, takes jupyter-book and nbconvert out of uv.lock. Bumping it here would be undone a commit later.

So after merge the count should drop to 2: diskcache #1, which has no fixed release, and nbconvert #92, which #227 removes. After #227 it should be 1 (diskcache).

pyproject.toml is unchanged. policyengine, policyengine-us, policyengine-uk and policyengine-core do not move, so reference outputs, scores and every published number stay where they are.

What changed in uv.lock

Each package moved with uv lock --upgrade-package <name>.

Package	Alerts (severity)	Locked	First patched
anyio	#72 (critical), #71	4.12.1 → 4.15.1	4.14.2
jupyter-server	#45 (critical)	2.19.0 → 2.21.1	2.20.0
aiohttp	#64 (high), #62, #63	3.14.1 → 3.14.5	3.14.3
cryptography	#65, #74 (high), #73	48.0.1 → 50.0.2	50.0.0
fsspec	#82 (high)	2026.2.0 → 2026.9.0	2026.6.0
mistune	#49, #50, #52, #55 (high), #51, #53, #54, #56, #57, #58	3.2.1 → 3.3.4	3.3.0
msgpack	#46 (high)	1.1.2 → 1.2.3	1.2.1
pyasn1	#59, #60, #61 (high)	0.6.3 → 0.6.4	0.6.4
soupsieve	#47, #48 (high), #69, #70	2.8.3 → 2.10	2.9.0
tornado	#68, #77, #79 (high), #67, #78, #66	6.5.7 → 6.5.10	6.5.9
urllib3	#80, #81 (high), #76	2.7.0 → 2.8.0	2.8.0
bleach	#43, #44; #42 (no patched release)	6.3.0 → 6.4.0	6.4.0
litellm (direct)	#75	1.88.1 → 1.88.6	1.88.6
multidict	#83	6.7.1 → 6.9.1	6.9.1
pytest (dev)	#20	8.4.2 → 9.1.1	9.0.3
diskcache	#1 (no patched release)	5.6.3 (unchanged)	none

Two packages move only because upgraded packages require them: typing-extensions 4.15.0 → 4.16.0 (anyio 4.15.1 needs typing_extensions>=4.16.0 below Python 3.15) and yarl 1.22.0 → 1.25.1 (aiohttp 3.14.5 needs yarl>=1.25.1). Nothing else in the lock changes: no source, hash-set or marker changes outside these 17 entries.

Three choices kept the bump narrow:

• litellm takes the 1.88.x backport, 1.88.6, not the 1.105 line. The fix for GHSA-3cv6-jpf6-8222 was backported to 1.88.6. Every synchronous benchmark request goes through litellm's completion/responses (policybench/eval_no_tools.py); batch runs call the provider SDKs directly (policybench/batch_eval.py). A jump of 17 minor versions would change provider adapters under future runs; that belongs in its own PR. The advisory itself concerns the litellm proxy server, which PolicyBench does not run.
  • What 1.88.6 changes for PolicyBench. I diffed the installed 1.88.1 and 1.88.6 packages:
    • llms/vertex_ai/gemini/, llms/gemini/, main.py, responses/, litellm_core_utils/prompt_templates/factory.py and litellm_core_utils/llm_response_utils/ are byte-identical.
    • utils.py adds only a Bedrock Mantle responses route.
    • Four files differ under llms/anthropic/:
      • chat/transformation.py changes only how sampling parameters are sent. temperature other than 1, top_p and top_k now go through _apply_sampling_param, which refuses them, or drops them under drop_params, for a model whose map entry sets supports_sampling_params: false (or, with no flag, whose name is a Fable or Opus 4.7/4.8 one). PolicyBench doesn't set LITELLM_LOCAL_MODEL_COST_MAP, so litellm loads its model map from GitHub at import. That map flags nine of PolicyBench's Claude ids: Fable 5 and 5.1, Opus 4.7, 4.8, 5 and 5.5, Sonnet 5 and 5.5, and Haiku 5.5. The bundled fallback map flags Fable 5, Opus 4.7 and Opus 4.8, and the name rule adds Fable 5.1.
      • common_utils.py adds those helpers and changes how _is_adaptive_thinking_model reads the map (it now also strips Bedrock and Vertex prefixes). The adapter consults it only when a request sets reasoning_effort, and no PolicyBench request sends one to Claude (REASONING_EFFORT_OVERRIDES is empty).
      • cost_calculation.py is the web-search cost change below.
      • experimental_pass_through/messages/interceptors/advisor.py gates caller-supplied advisor credentials, in the messages pass-through that PolicyBench doesn't use.
      • The bundled map gains a claude-fable-5 entry with the input, output and cache prices PolicyBench registers for it.
    • Request bodies are unchanged. I captured the HTTP body litellm builds, under each version, for every Claude id in policybench.config.MODELS (the board request with and without explanations) and for claude-haiku-4-5 and claude-haiku-4-5-20251001 at temperature=0.0, the request the reference explainer and five audit scripts send. All 24 bodies are identical between 1.88.1 and 1.88.6, with the remote map and with the bundled one. Neither the benchmark requests nor the tool pilot send sampling parameters, and Haiku 4.5 is unflagged in both maps, so its temperature=0.0 is still forwarded.
    • Cost code reads server_tool_use.web_search_requests from a dict as well as an object, and streamed chunks coerce it.
  • The tool pilot's Gemini patch. scripts/tool_pilot_argon.py (from Ready the tool pilot's v2-pilot run: prompt fixes, a no-tools arm, web timeout #239) patches litellm's Gemini search-tool handling and refused to run on any litellm but 1.88.1, so this bump would have stopped its Gemini web arms (review r2). Since the adapter it patches is byte-identical in 1.88.6, GEMINI_SEARCH_PATCH_LITELLM now names both releases. test_the_gemini_patch_keeps_search_beside_functions asserts the installed litellm is one of them instead of skipping, and runs on 1.88.6: litellm drops googleSearch without the patch and keeps it with the patch. A new test checks that the litellm in uv.lock is one of them, so a later lock bump fails CI rather than a live web arm. With the gate left at 1.88.1 only, both tests fail. Three other notes about litellm 1.88.1 behaviour (policybench/config.py, docs/runbook.md and the pilot's deferred-result note) rest on code that is unchanged in 1.88.6, so they now say 1.88.x.
• multidict stays on 6.x (6.9.1) instead of the 7.1.0 major that uv lock --upgrade-package multidict picks by default.
• soupsieve stays on 2.x (2.10). Unconstrained, uv splits it into 2.10 and 3.0.3 by Python version.

No constraint blocked a patch. (policyengine-core 3.32.29 allows pytest<10, so pytest 9 needs no override.)

The two alerts with no patched release

diskcache, GHSA-w8v5-vhqr-4h9v (#1): reachable, needs local write access, kept

• How it gets in. diskcache now enters only through litellm's caching extra (litellm[caching] in our dependencies). policyengine 6.1.2 no longer requires it (its METADATA has no diskcache line), and no other installed package imports it.

• The path is reachable. policybench/cli.py calls policybench.cache.enable_cache() for eval-no-tools, eval-no-tools-repeated and eval-no-tools-chunked, and for retry-failed-responses / repair-failed-rows with --use-cache. That sets litellm.cache = Cache(type="disk", ...), which wraps diskcache.Cache(dir) (litellm/caching/disk_cache.py). diskcache/core.py Disk.fetch unpickles any row stored as MODE_PICKLE. The mode is read from the cache's own sqlite file, so whoever can write the cache directory decides what gets unpickled.

• What an attacker needs. A way to write or replace what diskcache opens. The default directory is .policybench_cache, resolved against the process's current directory (policybench/cache.py), so it sits wherever the command was launched, not necessarily inside the checkout. Its permissions are whatever existed or the operator's umask produced. They need not match the source tree's, so write access to the cache does not imply write access to policybench/ or .venv/. An attacker could get in through:

  • the cache directory or any file under it (cache.db and its journals, the hex subdirectories, the .val value files);
  • group write as well as world write;
  • ACL grants that the mode bits don't show;
  • a writable parent, which lets someone rename the cache away and put their own in its place.

  These scenarios are inferred from the permission model; what is verified is that a row diskcache reads with MODE_PICKLE reaches pickle.load.

• Where it runs in CI. CI starts diskcache only in private temporary directories: tests/test_supervisor.py::test_supervised_run_keeps_policyengine_out_of_workers starts real policybench run workers (each runs policybench.cli eval-no-tools, which calls enable_cache()) with POLICYBENCH_CACHE_DIR under pytest's tmp_path. Provider calls there are faked, so CI exercises cache setup, not real provider responses stored and replayed. tests/test_cache.py patches Cache. The Vercel build is Bun-only.

• The residual risk is running with a cache directory, or a parent of it, that another account can write or replace. Trusted cache contents and safe filesystem permissions are assumptions of every eval run until a guard enforces them.

• Why not drop it. Removing diskcache means dropping the caching extra, and enable_cache() would then raise (litellm requires litellm[caching] for disk caching). Eval runs would lose the response cache that keeps resumed runs from re-billing provider calls.

• Decision. Keep it and document it here. Refuse a LiteLLM disk-cache directory another account could write (GHSA-w8v5-vhqr-4h9v) #226 (in review) adds a guard to enable_cache(). On Linux and macOS it refuses a cache when another account could write the cache or anything in it, or could replace the cache through a parent directory. A sticky parent such as /tmp is accepted, since only an entry's owner can rename it there. It judges this by ownership, permission bits, ACLs and the umask. It does not check Windows. That change touches the runner, so it is its own PR.

bleach, GHSA-g75f-g53v-794x (#42): not reachable; 6.4.0 is still affected

• 6.4.0 does not fix it. The advisory covers bleach.linkify(..., parse_email=True) and lists the range as = 6.3.0 with no fix. I diffed bleach/linkifier.py between 6.3.0 and 6.4.0. handle_email_addresses is identical, and the email regex changed only in formatting ("""...""".format(...) line wrapping). The advisory's PoC, bleach.linkify(("a." * 15000) + "a", parse_email=True), takes 6.40 s on 6.3.0 and 5.71 s on 6.4.0, against 0.034 s with parse_email=False on both. Dependabot will likely close Scoring rename: Approximate -> Continuous; document Within 1% in paper #42 after this bump because 6.4.0 is outside the stated range. Read that as a range artifact, not a fix. Take jupyter-book out of the paper render environment and uv.lock #227 then takes bleach out of the lock.
• The path is unreachable here. bleach enters via jupyter-book → jupyter-server → nbconvert[css] in the docs extra. In the installed environment, only nbconvert/filters/strings.py and nbconvert/preprocessors/sanitize.py import bleach, and both call only bleach.clean. No package calls bleach.linkify or uses Linker/LinkifyFilter (tornado's linkify is its own implementation). Quarto's Jupyter engine (share/jupyter/*.py in Quarto 1.9.36) imports neither nbconvert nor bleach. PolicyBench code never imports bleach.
• Why not drop it here. jupyter-book is the builder for the operational-docs site in docs/ (docs/myst.yml, docs/requirements.txt). Take jupyter-book out of the paper render environment and uv.lock #227 takes it out of the docs extra, leaving docs/requirements.txt as its only declaration, and so takes jupyter-server, nbconvert, bleach, mistune and soupsieve out of uv.lock: 18 of the 44 alerts open on 2026-10-14, plus nbconvert scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92. That is a docs-toolchain change, so it is its own PR.

Verification

I ran everything at the lock commit (0b4d91a1). After main's release #221 re-rendered the paper, I merged main in (head f7096c23) and re-ran the render comparison there. On 2026-10-25 I merged main again (ab82d1a5) for #238's certifi 2026.10.19. That merge took main's lockfile revision 5, written by uv 0.13.4, and main's pexpect entry; it changed no other package. Every run used a venv synced from this lock (uv sync --frozen --extra dev --extra docs; CPython 3.14.4 on macOS arm64). The logs are kept outside the repo.

• Full test suite: OPENBLAS_NUM_THREADS=1 uv run --frozen pytest -q gives 4606 passed, 51 skipped, exit 0 (43m46s) at 0b4d91a1. The local full run at f7096c23 was cut off at 82% with no failures when its session restarted, and I did not repeat it. CI's suite (pytest -m "not slow", Python 3.12, uv sync --locked) passed at f7096c23 and again at ab82d1a5 (run 40149555608), and is the check of record at the current head. The 14 warnings all come from packages this PR doesn't move: a divide warning in policyengine-us, policyengine-core's dpath.util deprecation, and pandas dtype notices. None are pytest 9 deprecations.

• Lint: ruff check . passes; ruff format --check . reports 163 files already formatted.

• Lock: uv lock --locked passes, and at the current head passes under both uv 0.13.4 (the version that wrote main's lock) and 0.13.5. A field-by-field comparison with main's lock shows changes only in the 17 entries listed above; every same-version entry and the lock metadata are identical.

• Paper: I re-rendered with uv run --frozen python paper/render_paper.py (HTML and PDF, Quarto 1.9.36) and compared the output with the committed app/public/paper/:

  • 17 of 19 files are byte-identical.
  • web/index.html differs on one line, a Quarto per-render cell id (<div id="00dcfe5d" class="cell" ...> → id="1bf227ac"). Its visible text is identical.
  • policybench.pdf has identical text (pdftotext -layout) and the same size (260,225 bytes). Its 66 differing bytes all sit in CreationDate/ModDate and the trailer /ID; with those masked, the files are byte-identical.

  At the merged head f7096c23, compared against Add Gemini 4 Argon and Muse Spark 1.3 (release, 50 models) [WIP] #221's committed paper, the result is the same: 17 of 19 files byte-identical; one HTML line differs, again a cell id (7f7f2048 → 4f934fb6); visible text and PDF text are identical; and the PDF (260,940 bytes) is byte-identical once its dates and /ID are masked.

  So the paper doesn't move. I restored the committed render, left the rendered_paper_artifacts pins alone, and did not run freeze_snapshot.py --rendered-only.

• App: cd app && bun run prepare-data && bun test && bun run lint && bun run build: 173 pass / 0 fail, lint exit 0, build exit 0, with a clean worktree afterwards.

• CI on this head: lint, test, paper (hermetic HTML render), app and the Vercel preview all pass.

Reviews

• r1 (GPT-6.1 Sol, at f7096c23): APPROVE, with three body findings, answered at ab82d1a5: the diskcache threat model, where CI runs diskcache, and a placeholder.
• r2 (GPT-6.1 Sol, at ab82d1a5): REQUEST_CHANGES. It had one blocking finding: the tool pilot's Gemini patch refused litellm 1.88.6, and its test skipped the mismatch. It also had two body findings: 53 open alerts, not 54, and Refuse a LiteLLM disk-cache directory another account could write (GHSA-w8v5-vhqr-4h9v) #226's coverage stated too broadly. All three are answered at ce3a83bc (above).
• r3 (GPT-6.1 Sol, at ce3a83bc): APPROVE. It traced completion through to the Gemini mapping and found both patched methods unchanged, and it found the alert arithmetic right. It had two non-blocking body findings, both answered above. First, the Anthropic adapter changes more than sampling parameters. Checking that also showed that the earlier list of no-sampling models was incomplete, so the bullet now gives the full list and the cross-version request-body comparison. Second, Refuse a LiteLLM disk-cache directory another account could write (GHSA-w8v5-vhqr-4h9v) #226 accepts sticky parents such as /tmp.

Invariants

This PR changes locked third-party versions, the tool pilot's litellm gate and its tests, and two comments. The properties it must preserve, and how each was checked:

• The scoring inputs do not move: policyengine, policyengine-us, policyengine-uk and policyengine-core keep the exact lock entries from main (lock diff).
• The published paper does not change: a re-render matches the committed HTML/PDF text, and the committed bytes and the manifest pins are untouched.
• uv sync --locked accepts the lock (uv lock --locked passes; CI syncs with --locked).
• The litellm in uv.lock is one the tool pilot's Gemini patch was checked against (test_the_locked_litellm_is_one_the_gemini_patch_was_checked_against).

axiom: n/a: dependency security bump, no policy logic

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 25, 2026 5:12am UTC

[MaxGhenis]

Taking this PR over from session local_90a11ea4, which has been idle since 2026-10-14; I've messaged it. Review r1b (Sol) approved at f7096c2 with three body nits; they're fixed in the body now, main is merged in (ab82d1a), and review r2 is queued. I'll merge it on green CI and an agreeing independent review, then remove its worktree. — session local_a8447b8d ("Finish five stale PolicyBench draft PRs")

[MaxGhenis]

Merged by the adopting Claude session (299e8a90), squash 555a0f2c, at reviewed head ce3a83bc.

Gates at merge:

• gh pr checks exited 0 on the latest runs at ce3a83bc (test, lint, app, paper, Vercel); rechecked after marking ready, and no new runs started.
• mergeable: MERGEABLE; no CHANGES_REQUESTED review.
• Independent review: GPT-6.1 Sol r3 at ce3a83bc, APPROVE. Its two non-blocking findings concerned the body only, and both are answered there: the Anthropic adapter bullet now lists all four changed files and the full set of no-sampling models, and the Refuse a LiteLLM disk-cache directory another account could write (GHSA-w8v5-vhqr-4h9v) #226 sentence notes that sticky parents are accepted. The head did not change.
• Added for r3's first finding: every Claude request body PolicyBench builds (22 board requests and the claude-haiku-4-5 temperature=0.0 request) is byte-identical between litellm 1.88.1 and 1.88.6, under both the remote and the bundled model map.

Next: #227 relocks on this lock and removes nbconvert. The bleach alert (#42) closes only because 6.4.0 falls outside the advisory's range, while 6.4.0 is still affected (see the body).

🤖 Generated with Claude Code

[MaxGhenis]

Post-merge alert check (Dependabot's scan of 555a0f2c, 07:59Z):

• All 50 alerts this PR targeted are now fixed, including bleach Scoring rename: Approximate -> Continuous; document Within 1% in paper #42, which closes because 6.4.0 is outside the advisory's range even though it is still affected.
• As predicted, two alerts this PR couldn't fix stay open: diskcache Initial version #1 (no patched release; Refuse a LiteLLM disk-cache directory another account could write (GHSA-w8v5-vhqr-4h9v) #226 adds the guard) and nbconvert scenario_121 encodes disability in inputs PE does not map to USDA disability, zeroing SNAP #92 (Take jupyter-book out of the paper render environment and uv.lock #227 removes nbconvert from the lock).
• The same scan opened three new filelock alerts, Add dataset version selector (v1.0 / v1.1) #93 (high), Document per-model reasoning configuration in the manuscript #94 and Unpin GPT-5.5 reasoning effort for uniform unconfigured defaults #95 (low). Their advisories (GHSA-v4x8-3wr5-3wq8, GHSA-567q-5qvw-pf92, GHSA-9vhv-jxww-xfm5) were published 2026-10-23, after this PR's inventory. filelock is 3.24.3 both before and after this PR (pulled in by huggingface-hub), so the PR didn't cause them. The patched release is 4.2.1, and they need their own PR.

🤖 Generated with Claude Code