fix(core): measure the structure filter from the #1, not the population minimum (#353)

[send]

Summary

rerank's structure filter measured its threshold from the population's minimum structure cost + structure_cost_filter. The population depends on the oversample (1-best 10/30, N-best n×3 / 60), so a cheap-transition path that never wins (カナ|や shape) could lower the threshold and drop the paths that do win (#353 symptom 1; four earlier symptom fixes — prefix floor, transition cap, identity exemption, 4000→6000 — all patched this).

The threshold is now measured from the anchor = the pre-history #1 (the cheapest path after features, the same path cost-gap admission and Numeric measure from): anchor sc (single segment imputed at the prefix floor) + structure_cost_filter.

• The Add IME app with romaji-to-kana and Rust FFI bridge #1 can no longer be dropped (by construction), and the threshold depends on the population only through the Add IME app with romaji-to-kana and Rust FFI bridge #1.
• On the same population, every path that survived before still survives (threshold ≥ old one). structure_cost_filter keeps its key and default; a user-set value now means the same or looser (noted in SPEC / default_settings).
• rerank returns the StructureFilter it used and reports each drop to on_drop; explain shows them (dropped_by_structure, structure_filter; text Dropped by structure).
• tune's copy of the filter (hard_filter) is removed: tune scores top-1 only, and the filter can no longer touch the top-1.

Closes #353. Its symptom 2 (1-best vs N-best head disagreeing) turned out not to come from the filter — the two known cases persist with the filter off — and is tracked in #361 (PR-E2: argmin over different oversamples).

Measurements

Frozen commit-log snapshot: 7085 lines / 505 rank>0 selections / 469 distinct readings. Corpus readings: 142. Arms: main (450fa55), main with the filter disabled (oracle, structure_cost_filter = 10^9), this PR — all at the same time. Counts only.

	main	filter off	this PR
accuracy-corpus	109/109 (28 skip)	109/109	109/109
accuracy-corpus-history	11/11	11/11	11/11

• Corpus readings (142), top-1 at n=1 / n=20 / candidate list, with and without history: 0 changes vs main and vs the oracle. n=1 ≠ n=20: the same 2 known readings (1-best vs N-best top-1 mismatch comes from argmin population width, not the structure filter (PR-E2) #361).
• Page 1 (first 9, n=20, 142 readings): 5 readings change; per reading the number of junk slots does not grow (net ≤ 0), and にしたほうがいい gains にしたほうが良い (the structure filter threshold depends on the oversample population (drops real paths, flips top-1) #353 symptom-1 example). Returned slots are mostly one-kanji fragments (確認師ますね …), which displace equally junk ones (確認します値 …) — that mechanism is PR-F's (Enable single_char_kanji_penalty: single-kanji content fragments on page 1 (PR-F) #362), not this PR's.
• commit-log readings (469), top-1 vs main: 6 without history (n=1: 1), 1 with history — every one equals the oracle's Add IME app with romaji-to-kana and Rust FFI bridge #1. None turns a committed (learned) Add IME app with romaji-to-kana and Rust FFI bridge #1 into an unlearned one; the history case is learned vs learned, the cheaper one wins (as designed in PR-G).
• replay, no history, vs main: lost 1 / demoted off page 0 / demoted within page 5 (all: a returned path entered above) / improved 2 — within the pre-registered bounds (≤ 1% / ≤ 2%) and no worse than the oracle (1 / 0 / 7). With history: lost 0.
• auto-commit: over 2318 typing prefixes of those readings, whether auto-commit can fire never changes; the committed first segment changes for 1 prefix (no history; matches the oracle) and 0 with history.
• Cost-gap T re-evaluation (trigger: "structure filter threshold depends on the oversample population (drops real paths, flips top-1) #353 lands"): (b1) lost + off-page 3 / 1 / 1 / 1 at T = 6000 / 8000 / 10000 / 12000; (b2′) bin below T 19 / 2 (≤ 2.5); (b3) both corpora pass only at 8000 → T stays 8000 (recorded in DEFAULT_CANDIDATE_MAX_COST_GAP).
• Speed (5 inputs × 200, 7 runs, min / median vs main): 1-best +1.1% / −3.1%, 1-best with history +0.7% / −1.9%, n=20 −2.8% / +0.9%.

dropped_by_structure over the 142 readings: 593 paths in 67 readings; 26 within the cost-gap bound, none of them in page-1 range.

Known follow-ups

• The filter now looks dead under admission. The pre-registered deletion check holds on all three conditions (page 1 identical to the filter-off oracle on 142/142 readings, no difference in which candidates replay selects, no filter-only drop in page-1 range). By that rule the filter (and its prefix floor / transition cap / imputation) goes in a separate PR; the API added here (StructureFilter, on_drop, explain fields) is provisional until then.
• The width disagreement can also go the other way now: a price argmin only a large population sees is no longer dropped by chance. Measured on the 469 readings: 0 new, 3 resolved. Tracked with 1-best vs N-best top-1 mismatch comes from argmin population width, not the structure filter (PR-E2) #361.

Test plan

• mise run test (lint + workspace tests), accuracy ×2, read-pin, task-screen, check-sources + audit, compile-swift + test-swift, msrv 1.88.0
• Tests that fail under the old rule: the threshold depends on the population only through the Add IME app with romaji-to-kana and Rust FFI bridge #1, the Add IME app with romaji-to-kana and Rust FFI bridge #1 is never dropped, 1-best == N-best head when only the filter differed, tune top-1 == rerank index 0
• Mutations caught: no prefix floor / anchor by Viterbi cost / identity excluded from the anchor
• Measurements above (frozen snapshot, all arms at the same time)

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-01T11:03:33.763590Z	3cae806	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[send]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 3cae806ecf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[send]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 3cae806ecf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".