fix(learn): honest ranking metrics, seed-sensitive generators, distinct benchmark configs

.docs/learned-heuristics.md

@@ -96,7 +96,10 @@ means nothing. `rank_weight=0.8` splits the objective; 0.8 is a reasonable
 default, not a tuned optimum.
 
 The metric we select checkpoints on is **top-1 ranking accuracy**, not
-validation MAE, for the same reason.
+validation MAE, for the same reason. It is measured on the held-out instances,
+and a tie with `k` siblings earns `1/(k+1)` — its chance of winning a random
+tie-break — so a model that scores every state alike rates as guessing, not as
+perfect.
 
 ### Not admissible, and it does not pretend to be
 
@@ -112,7 +115,8 @@ become unsound.
 Train on a small ladder, test on instances two to three times larger with
 different seeds. Planner is `gbfs` throughout, budget 30 000 expansions /
 30 s. Coverage is the fraction solved; expansions and seconds are means over
-solved instances.
+solved instances — a convention that flatters low coverage, and that the code
+no longer uses (see [Corrections, 2026-11-03](#corrections-2026-11-03)).
 
 ### blocksworld — trained on 3–6 blocks, tested on 9–13
 
@@ -126,8 +130,8 @@ solved instances.
 
 Against `hff`: **3.8× fewer expansions, 15× faster, and slightly cheaper plans.**
 Corpus construction took 0.5 s, training 0.1 s, search-cost tuning about 50 s.
-Held-out MAE 0.92, top-1 accuracy 0.923. The model has 1025 parameters over 14
-features.
+Held-out MAE 0.92, top-1 accuracy 0.923 (the top-1 was not held out; see the
+corrections below). The model has 1025 parameters over 14 features.
 
 The tuned row is what this prints, end to end, in about a minute:
 
@@ -161,7 +165,8 @@ a correction to an earlier claim about *why* the tuned model is better, are in
 The cleanest win: half the expansions of `hff`, 5.6× faster, and *better plans*.
 Top-1 accuracy 1.000 — gripper's structure is almost entirely captured by "how
 many balls are still in the wrong room", which is exactly what these features
-encode.
+encode. (As measured here that 1.000 was not evidence of anything, and this
+table was built from duplicated instances; see the corrections below.)
 
 Note `blind`'s plan cost of 25.0, the lowest in the table. It solved only 40% of
 the instances and its search is effectively breadth-first on those, so it finds
@@ -180,6 +185,63 @@ first column.
 **A clear loss.** `hff` expands 6× fewer nodes and finds plans 42% cheaper. The
 learned heuristic beats `goalcount` and nothing else.
 
+### Corrections, 2026-11-03
+
+A review of `jupyddl.learn` found bugs that touched figures on this page. The
+figures above are left exactly as measured — they match
+`.docs/assets/rl-data.json`, the Research view and the video — and this is what
+each one actually measured.
+
+- **Top-1 accuracy was neither held out nor tie-safe.** It was computed over
+  every ranking group in the corpus, training instances included, while
+  `jupyddl learn` printed it as held out; and a sibling that scored level with
+  the successor the plan took counted as a correct ranking. Checkpoint
+  selection read the same number. The tie rule can only push it up, and
+  training groups usually do. Re-run with the fix — same corpora, seeds and 60
+  epochs, top-1 over the held-out instances' groups only, a tie with `k`
+  siblings credited `1/(k+1)` — which takes seconds:
+
+  | | published | re-measured | held-out groups |
+  |---|---|---|---|
+  | blocksworld | 0.923 | 0.929 | 28 of 101 |
+  | logistics | 0.656 | 0.667 | 42 of 157 |
+  | gripper, old generator, two seeds per size | 1.000 | 1.000 | 30 of 80 |
+
+  The old code reproduces each published figure on these corpora. Gripper's
+  protocol is not recorded beyond the sizes, so its row uses `jupyddl learn`'s
+  default of two seeds per size.
+
+  The values barely move, but the checkpoint does: blocksworld now keeps epoch
+  12 rather than 20, and its held-out MAE is 1.16 rather than 0.92; logistics
+  keeps epoch 22 rather than 9. **Every other learned-heuristic figure on this
+  page and in [rl-for-search.md](rl-for-search.md) was measured on the
+  checkpoint the old rule kept.** They are true of that model; the fixed code
+  trains a different one, and none of them has been re-measured.
+- **Gripper's 1.000 could not have come out any other way.** In that corpus, 59
+  of the 80 groups held a sibling identical in features and step cost to the
+  successor taken — picking a ball up with the other hand gives a different
+  state that the features cannot tell apart — so a model that outputs a
+  constant also scored 1.000. Worse, the gripper generator ignored its seed:
+  the "held-out" instances were copies of training ones, and since `--evaluate`
+  draws two seeds per size, the ten test instances in the table were five,
+  each counted twice. The means are unchanged by that, but rest on half the
+  sample claimed; a `--cem` run on gripper would also have tuned and selected
+  on identical instances. With identical siblings dropped, top-1 is still 1.000
+  on the old instances and on 27 groups of genuinely unseen ones from the
+  fixed generator, so the claim survives; the published number was simply not
+  evidence for it. The table cannot be re-run as it stands, because the same
+  command now generates different instances.
+- **Means over solved runs flattered low coverage.** The imitation-only
+  blocksworld row's 366 averages the nine instances it solved and drops the one
+  it could not finish in 30 000 expansions; its plan cost of 49.1 is over the
+  same nine. Charging that failure as `search_cost` does, at twice the budget,
+  its mean is 6 329. On the nine instances every configuration solved, the
+  tuned model averages 129, not 137 — computed from the per-instance table in
+  [rl-for-search.md](rl-for-search.md). `blind`'s 20 198 on gripper is the same
+  flattery, a mean over the instances it happened to solve; not re-measured.
+  `summarise_transfer` now charges failures that way and reports plan cost
+  only over instances every configuration with any coverage solved.
+
 ## Why logistics fails, precisely
 
 This is the most useful result in the set, because the mechanism is exact

.docs/rl-for-search.md

@@ -133,7 +133,9 @@ on a small instance.
 | after CEM | **100** | **137 expansions, 1.00 coverage** |
 
 CEM fixed the coverage failure imitation left behind, then shaved expansions:
-2.7× fewer nodes on instances no stage of training ever saw.
+2.7× fewer nodes on instances no stage of training ever saw. (The two means are
+over different instance sets; see [the second
+correction](#a-second-correction-2026-11-03).)
 
 ### A correction, and what the numbers actually say
 
@@ -203,6 +205,38 @@ The general lesson is the ordinary one, which the derivative-free framing made
 easy to forget: **change one thing at a time, and look at the distribution
 before believing the mean.** `learn_heuristic` always passes one.
 
+### A second correction, 2026-11-03
+
+A review of the learning code turned up three more things that bear on this
+page. Nothing above has been re-run — the expensive measurement is not repeated
+for a correction note — so the figures stand as measured, and this says what
+they measured.
+
+- **"366 expansions, 0.90 coverage" is a mean over the nine instances
+  imitation solved.** The tenth, `blocksworld-13-7777`, contributed nothing
+  rather than 30 000, and the 137 it is compared with averages all ten. On the
+  nine both solved, the per-instance table gives 129 for σ=0.15, so the
+  like-for-like improvement is 366 → 129, 2.8× rather than 2.7×; charging the
+  unsolved instance as `search_cost` does (twice the budget) puts imitation's
+  own mean at 6 329. The bias ran in imitation's favour, which is the
+  dangerous direction: it is how a heuristic that gives up looks cheap.
+  `summarise_transfer` averaged over each heuristic's own solved runs; it now
+  charges failures and reports coverage beside every mean.
+- **Every model here descends from a checkpoint chosen by a broken score.**
+  Imitation kept the epoch with the best top-1 measured over *training*
+  instances, with a tie counted as a correct ranking. Re-run with the fix on
+  the same corpus, blocksworld keeps epoch 12 rather than 20 (details and the
+  re-measured top-1 in [learned-heuristics.md](learned-heuristics.md#corrections-2026-11-03)).
+  The CEM history, the per-instance table, the σ comparison and the headroom
+  measurements below are true of the model the old rule kept; the fixed code
+  starts CEM from a different one. Not re-measured.
+- **DAgger's retraining split leaked.** Relabelled states are filed under
+  `X#dagger1`, and the split keyed on the full name, so a training instance's
+  own relabelled states could sit in validation. That fed only the MAE
+  tie-break in checkpoint selection, not the search costs quoted (89 → 15 and
+  89 → 164, which `search_cost` measures directly), but it made the validation
+  MAE of those rounds optimistic. The split now keys on the name before `#`.
+
 ## The three things that decide whether this works
 
 ### Start from the imitation solution

CHANGELOG.md

@@ -141,6 +141,60 @@ plans because it replays them through the same model. They are now pinned by
 - `black` is pinned to one major version in both the dev extra and the
   `format` workflow, which installed whatever was newest; Renovate now pins
   third-party actions to commit digests.
+- **Learned-heuristic top-1 was neither held out nor tie-safe.** A sibling
+  scoring level with the successor the plan took counted as a correct ranking,
+  so a model that outputs a constant scored 1.000 — and checkpoint selection,
+  which reads top-1, could prefer a collapsed epoch. `Corpus.split` also gave
+  the held-out side no ranking groups, so the "held out" top-1 `jupyddl learn`
+  printed was measured over training instances. A tie with `k` siblings now
+  earns `1/(k+1)`, a sibling identical to the chosen successor in features and
+  step cost is not recorded at all, and selection and the reported top-1 use
+  the held-out groups. The published figures came from the old rule; see the
+  dated corrections in `.docs/learned-heuristics.md` and
+  `.docs/rl-for-search.md`.
+- **A DAgger or bootstrap state could validate against its own instance.**
+  `split` treated `X#dagger1` as an instance apart from `X`; it now groups on
+  the name before the `#`.
+- **`summarise_transfer` averaged each heuristic over the runs it solved**, so
+  one that solved only the easy instance looked many times better than one that
+  solved both. An unsolved run is now charged twice the budget, as
+  `search_cost` charges it, and plan cost compares only instances every
+  configuration with any coverage solved. The imitation-only blocksworld mean
+  the docs quote as 366 is 6 329 under that rule.
+- **A malformed learned heuristic loaded, then gave one answer with NumPy and
+  another without.** A network whose input width is not the feature vector's,
+  a ragged row, a `sizes` field that does not describe the weights, a
+  non-finite value or a non-positive scale is now a `ValueError` on load.
+  `MLP.set_flat` checks the length before writing anything, instead of leaving
+  a wrong-length vector half-applied.
+- **Every negated fact shared one feature, `not`.** Complement facts now key on
+  the predicate they negate (`not-sampled`), so rovers and workshop get a slot
+  per negated predicate. A model trained on a domain with negative
+  preconditions before this sees different features now; retrain it.
+- `train()` with a pure ranking objective and no ranking groups returned its
+  random initialisation as a trained model; it now raises. `learn_heuristic`
+  spent a generator passed as `sizes` on its first read, recording `[]` and
+  raising under `cem_iterations`; size arguments are now read once, on entry.
+- **`gripper` and `workshop` ignored their seed.** Twenty seeds gave one
+  instance, so a split by seed held out copies of training instances, and the
+  RL stage's tuning and validation sets were the same problems. The seed now
+  decides which room each ball and the robot start in (every ball still has to
+  change rooms), and how far along the pipeline each part starts. **Every
+  generated gripper and workshop instance changes**; `demos/workshop` is the one
+  in which every part starts raw, and `generate` no longer reproduces it.
+- **Generators could emit an instance solved before it starts.** `blocksworld`
+  compared tower lists in order and skipped the check for two blocks, where 75
+  of the first 200 seeds came out trivial; `random-strips` could draw a goal
+  that already held. Both now re-draw only in that case, so every instance that
+  was not trivial is byte-identical to before.
+- **The benchmark harness merged configurations that differed in planner
+  arguments** — `wastar` at weights 1 and 5 summarised as one entry with twice
+  the instances — and recorded `("gbfs", None)` as heuristic `""` although it
+  ran `hff`. Rows carry a `config` label with the arguments
+  (`wastar/hff(weight=5.0)`, also a new CSV column), which `summarize` and the
+  dashboard group on; a repeated config is suffixed `#2`; and `heuristic` is
+  what `solve_task` actually ran, worked out by the new
+  `jupyddl.api.resolve_heuristic`.
 
 ## [2.4.0]
 

README.md

@@ -471,12 +471,14 @@ and to make the difference between planners obvious:
 | `errands` | **preferences + constraints**: soft goals priced by a metric | 10 |
 | `timed-market` | **timed initial literals**: the plan waits for opening time | 4 |
 
-Two of them are byte-for-byte generator output — `blocksworld12` is
-`generate blocksworld --size 12 --seed 3`, `workshop` is
-`generate workshop --size 3 --seed 1`. The rest were either hand-written
+`blocksworld12` is byte-for-byte generator output:
+`generate blocksworld --size 12 --seed 3`. `workshop` was
+`generate workshop --size 3 --seed 1` until that generator's seed began to
+decide how far along the pipeline each part starts; it is the instance in which
+every part starts raw. The rest were either hand-written
 (`sokoban`, `hanoi`, `elevator`, `errands`, `timed-market`, and `network`,
 whose recursive axiom is the point of it) or generated and then edited by hand,
-so `generate` will produce something equivalent but not identical.
+so `generate` will produce something similar but not identical.
 
 The `pddl-examples` git submodule supplies the smaller instances used by the
 parser and grounder tests.

jupyddl/api.py

@@ -37,20 +37,30 @@ def solve_task(
     needs a heuristic means ``blind``.
     """
     planner = make_planner(search, **planner_kwargs)
-    heur = None
-    name = heuristic
-    if name == "none":
-        name = "blind" if planner.requires_heuristic else None
-    elif not name and planner.requires_heuristic:
-        # An optimal planner gets an admissible default: hff would quietly
-        # cost it its guarantee.
-        name = "lmcut" if planner.optimal else "hff"
-    if name:
-        heur = make_heuristic(name, task)
+    name = resolve_heuristic(planner, heuristic)
+    heur = make_heuristic(name, task) if name else None
     budget = make_budget(max_expansions, time_limit)
     return planner.search(task, heur, observer=observer, budget=budget)
 
 
+def resolve_heuristic(planner, heuristic):
+    """The heuristic :func:`solve_task` actually runs ``planner`` with.
+
+    ``planner`` is a planner instance; ``heuristic`` is what the caller asked
+    for. Returns a registry name or spec, the heuristic object passed in, or
+    ``None`` for no heuristic at all. Anything that records which heuristic
+    produced a result should ask this rather than echo the request, which for
+    ``None`` and ``"none"`` names something other than what ran.
+    """
+    if heuristic == "none":
+        return "blind" if planner.requires_heuristic else None
+    if not heuristic and planner.requires_heuristic:
+        # An optimal planner gets an admissible default: hff would quietly
+        # cost it its guarantee.
+        return "lmcut" if planner.optimal else "hff"
+    return heuristic or None
+
+
 def solve(
     domain_path: str,
     problem_path: str,