Repository navigation
feat(core): PR-G — learned surface is #1 (cheapest learned path first) - #363
Merged
Merged
Conversation
…-G1..T-G9, SPEC) History reranking moves the cheapest learned path (whole_path_boost > 0) to index 0; the Override stage leaves a learned index 0 alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… ScoredPath::single Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Numeric / Katakana paths are created after history reranking, so they were never learned: a stale learned surface stayed #1 over a number compound the user commits. The Override stage now boosts what it creates with the same helper as history reranking; a learned created path takes index 0 from an unlearned one or undercuts a learned one, and explain reports its boost. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
, stale docs Replay's gap is measured from the list's #1, which can now be a learned path priced above what follows: a negative gap is a re-correction and is counted apart (gap_below_top), the counter PR-G's post-ship revert trigger reads. Docs that still said the list is cheapest-first, that decay keeps stale history from dominating, or that a number compound always takes index 0 now state the learned-first rule; the 1-best oversample comment records why it stays at 30 (#361). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts limits §ブースト計算 now holds the rule once, with where it does not reach: learned surfaces outside the history-stage population (injected, #361 for the 1-best), user dictionary words, the kana rescue price, and what Fn+Delete falls back to. Pipeline, Rewriters and CostFunction lines point to it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ring T-G9′ (first-segment reading differs, session built with the history), T-G2′ (price order ≠ boost order), T-G5 (a)/(b) through the pipeline, T-G7′ (explain on a rotated list) and T-G8′ (おむろん → OMRON, gap 29060 over an 18000 boost) each fail with the rotate removed; the boost-reorder test is back to measuring boost size. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Override stage now inserts in cost order below a learned index 0 and then applies the same learned_first as history reranking (cheapest learned first, a replaced learned index 0 back at its price), instead of comparing each candidate with index 0 only. A duplicate's price owner is chosen on pre-history prices before the resolved path is boosted again, so a candidate with fewer segments no longer raises the path. Tests give the stage a boost as production does; stale comments and the replay label width fixed; SPEC notes that Rewriters can move the returned #1 to 3rd. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rride price A same-surface Override candidate is learned exactly when the path it duplicates is, so the frozen prefix (which keeps unlearned candidates off a learned index 0) no longer stops a learned one from repricing it; learned_first then settles #1. Before, only index 0 missed the compound price, and adding another learned path changed which learned surface won. Docs: replay's re-correction count needs the history frozen at the window start; SPEC states the auto-commit tracker follows the list's #1, the kana move below a learned #1, where decay applies, and that worst-based policy prices can split learned-vs-learned #1 between the 1-best and the list. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Owner
Author
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e366305ff5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A learned duplicate repriced below the learned index 0 was inserted at index 0 directly, so learned_first saw the cheapest learned already first and left the replaced learned #1 at index 1 (Codex R1). The re-gate then found the root: the frozen prefix was a position, and removing the head to reprice it froze the next path instead, making results depend on the rewriters' order. The stage now takes the held head (Model: always; Override: a learned index 0) out of the list while candidates go in, reprices it in place for a same-surface learned candidate, puts it back, and learned_first alone decides index 0. A displaced learned #1 goes back before equal prices, as Override inserts. Regression tests for the R1 case, rewriter-order independence and the tie rule each fail on the previous implementation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Owner
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Owner
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
PR-G (rank 2+ ゴミ候補プログラム)。候補列の #1 を「この読みで学習済みの表記」の中から選ぶ。
history_rerank_at): 価格順に並べたあと、学習済みの経路 (whole_path_boost > 0、ScoredPath::is_learned) のうち最安のものを index 0 に置く。残りは価格順のまま。学習済み同士の順は価格 (decay 込み) で決まるapply_history_boost) で学習ブーストをかける。学習済みの数詞複合語やカタカナは、学習済みの Add IME app with romaji-to-kana and Rust FFI bridge #1 と価格で争うreranker::learned_first: 最安の学習済みを index 0 に置き、置き換えられた学習済みの index 0 は価格の位置に戻す。学習ブースト段と Override 段の最後の両方で呼ぶreplay-commit-logに< 0 (learned #1)行を追加した。学習済みの Add IME app with romaji-to-kana and Rust FFI bridge #1 より安い表記を選んだ件数 (= 再訂正) で、出荷後の撤回条件の計数器になる動機: PR-E (#353) の計測で、学習済みの #1 が経路の価格差で未学習の表記に負ける退行が 3 件出た。PR-E の前提として PR-G を先に入れる。
宣言する挙動変更
計測 (凍結した commit-log 6448 行 / rank>0 468 / distinct 434 読み。個人内容は載せず件数のみ。main = eb15ead、PR-G = e366305)
< 00幅 (1-best ≠ 候補列 #1、履歴あり): hard rule の例外を 1 件受け入れる
速度
convert_1best/_history/_10best× 4 入力): 計測時にマシン負荷で ±30% 揺れ、実行順で符号が反転したため、上の交互計測を正とする出荷後の撤回条件 (事前登録)
--from-line) にし、その時点の学習履歴 (user_history.lxud+.wal) を凍結コピーする。現在の履歴を渡すと、選び直した表記も学習済みになって 0 件になるためreplay-commit-log --history <凍結コピー> --from-line <起点>で再生する< 0 (learned #1)の件数 = 固定された学習済みの Add IME app with romaji-to-kana and Rust FFI bridge #1 を避けて、それより安い (出荷時点で未学習の) 表記を選び直した件数。出どころは問わないTest plan
mise run test(lint + workspace tests)、cargo test -p lex-core --features trace,neural、msrv 1.88.0 checkmise run accuracy/mise run accuracy-historyhistory_puts_cheapest_learned_first/ T-G4learned_paths_order_by_price_not_by_boostviterbi_best_reinserted_below_learned_top/ (b)displaced_fragment_top_is_dropped_by_admissionlearned_surface_outranks_number_compoundtest_explain_paths_match_production_nbestauto_commit_and_stability_follow_learned_toplearned_number_compound_outprices_stale_learned_surfaceoverride_price_is_chosen_before_historylearned_candidate_undercutting_learned_top_demotes_it_to_its_pricelearned_index0_takes_its_compound_pricerepriced_learned_duplicate_demotes_the_learned_top_to_its_price(Codex R1)held_head_does_not_pass_to_the_next_pathdisplaced_learned_head_precedes_equal_prices🤖 Generated with Claude Code