bootstrap.py: a validator's exit 2 ends bootstrap REFUSED, exit 2, an unreadable upstream path, cache or pin is no answer and a reftable leg's cherry-pick or revert is read from its ref store, so no traceback or ended pick follows (#764)

docs/cli.md

@@ -44,7 +44,7 @@ one today, and the generator refuses to write this file if one ever appears.
 
 | tool | what it is for |
 | --- | --- |
-| [`templates/assembly-root/scripts/bootstrap.py`](#templatesassembly-rootscriptsbootstrappy) | The one command after `git clone --recurse-submodules`: places each leg on its tracking branch AT the pinned commit. What `make bootstrap` runs. |
+| [`templates/assembly-root/scripts/bootstrap.py`](#templatesassembly-rootscriptsbootstrappy) | The one command after `git clone --recurse-submodules`: places each leg on its tracking branch AT the pinned commit only where that loses no commit, and otherwise leaves its checkout where it is, saying why. What `make bootstrap` runs. |
 | [`templates/assembly-root/scripts/validate-manifest.py`](#templatesassembly-rootscriptsvalidate-manifestpy) | The project's own `project.yaml` validator: the manifest against the naming and path policies it was cut from. |
 | [`templates/assembly-root/scripts/validate-pins.py`](#templatesassembly-rootscriptsvalidate-pinspy) | The project's own LOCKSTEP validator: the gitlink, the pin and the workflow references must agree, and every digest must recompute. |
 | [`templates/family-root/scripts/bootstrap.py`](#templatesfamily-rootscriptsbootstrappy) | The holder's own bootstrap: mounts every member at its pin and runs each member's. |

scripts/render-cli-reference.py

@@ -180,8 +180,9 @@ class Tool(NamedTuple):
     Tool("templates/assembly-root/scripts/bootstrap.py", "python",
          "python3 scripts/bootstrap.py",
          "The one command after `git clone --recurse-submodules`: places each "
-         "leg on its tracking branch AT the pinned commit. What `make "
-         "bootstrap` runs.",
+         "leg on its tracking branch AT the pinned commit only where that "
+         "loses no commit, and otherwise leaves its checkout where it is, "
+         "saying why. What `make bootstrap` runs.",
          "scripts/bootstrap.py", INSTALLED_IN_ASSEMBLY_ROOT),
     Tool("templates/assembly-root/scripts/validate-manifest.py", "python",
          "python3 scripts/validate-manifest.py",

templates/assembly-root/scripts/bootstrap.py

@@ -67,9 +67,16 @@
       wallet-carried authority is not misconfigured, and a bootstrap that
       failed there would have made the layout load-bearing again.
 
-  (d) EXITS 0 unless a validator failed, and 2 where step (a) or the
-      upstream notice cannot start git, refused `git-not-runnable` as the
-      validators refuse it (#694).
+  (d) EXITS 0 unless a validator failed, and says which way. A validator
+      that FOUND something, any exit but 2, ends the run `bootstrap FAILED:
+      <name> (exit <code>)`, exit 1. A validator that REFUSED, its exit 2,
+      could not check at all, and the run ends `bootstrap REFUSED: <name>
+      (exit 2)`, exit 2 (#764). Where one validator found and another
+      refused, both lines are said, the finding first, and the run exits 2,
+      since a check that could not run outranks a finding. Step (a) and the
+      upstream notice refuse `git-not-runnable`, exit 2, where git cannot be
+      started, as the validators do (#694): the run stops there and says
+      neither line, though a validator may have found or refused by then.
 
 SCHEMA-NEUTRAL BY CONTRACT. A one-repository project runs the same command:
 step (a) is a no-op where there are no submodules, and steps (c) and (d) are
@@ -129,6 +136,13 @@
     ("CHERRY_PICK_HEAD", "a cherry-pick", "cherry-pick"),
     ("REVERT_HEAD", "a revert", "revert"),
 )
+#: The two `IN_PROGRESS` markers git writes as REFS (#764). On a leg whose
+#: refs are in a reftable (`git init --ref-format=reftable`, git 2.45 and
+#: later) neither is a file in the git dir while its operation is in
+#: progress: each is in the ref store, which step (a) asks as well
+#: (`_in_ref_store`). `MERGE_HEAD` is a file on both backends, as the two
+#: rebases' directories and `sequencer/todo` are (measured with git 2.47.1).
+IN_REF_STORE = ("CHERRY_PICK_HEAD", "REVERT_HEAD")
 #: What a cherry-pick or a revert of SEVERAL commits leaves while it is not
 #: done, asked after `IN_PROGRESS`: its stop committed by hand takes the
 #: head file away and leaves this, whose first word says which it is.
@@ -149,6 +163,9 @@
     ("manifest", ["scripts/validate-manifest.py"]),
     ("pins", ["scripts/validate-pins.py"]),
 )
+#: The exit code a validator ends a REFUSAL with: it could not check, which
+#: is not a finding (#764). Every other non-zero code is a failure.
+VALIDATOR_REFUSED = 2
 
 
 def _run(root: Path, args: list[str]) -> int:
@@ -672,9 +689,10 @@ def _on_a_branch(current: str | None) -> bool:
     return current is not None and current.startswith("refs/heads/")
 
 
-def _in_progress(sub: Path) -> tuple[str, str, str] | None:
+def _in_progress(sub: Path) -> tuple[tuple[str, str, str], str] | None:
     """The `IN_PROGRESS` row of the operation git has in progress in the
-    leg, or `None` (#704).
+    leg, with where its marker is (`git dir` or `ref store`), or `None`
+    (#704).
 
     Asked of the leg's own git dir, which `git rev-parse --git-dir` names
     (a submodule's is under the root's `.git/modules/`, a linked
@@ -683,6 +701,12 @@ def _in_progress(sub: Path) -> tuple[str, str, str] | None:
     path, so no name of a directory can fail the read. A leg git names no
     git dir for has nothing to read here, and the move after this says what
     git says of it.
+
+    A cherry-pick's or a revert's marker is asked of the leg's ref store
+    too (`IN_REF_STORE`, #764). On a reftable leg it is there and in no
+    file, so a leg stopped in either was moved, and the move ended it as
+    #704 measured on the files backend: `CHERRY_PICK_HEAD` or `REVERT_HEAD`
+    gone, and what it had staged carried onto the tracking branch.
     """
     try:
         said = git_out(["rev-parse", "--git-dir"], cwd=sub, binary=True)
@@ -691,8 +715,24 @@ def _in_progress(sub: Path) -> tuple[str, str, str] | None:
     gitdir = sub / os.fsdecode(said.rstrip(b"\r\n"))
     for row in IN_PROGRESS:
         if os.path.lexists(gitdir / row[0]):
-            return row
-    return _sequence_in_progress(gitdir)
+            return row, "git dir"
+        if row[0] in IN_REF_STORE and _in_ref_store(sub, row[0]):
+            return row, "ref store"
+    sequence = _sequence_in_progress(gitdir)
+    return None if sequence is None else (sequence, "git dir")
+
+
+def _in_ref_store(sub: Path, name: str) -> bool:
+    """Whether the leg's ref store holds the ref called exactly `name`
+    (#764).
+
+    `git show-ref --exists` (git 2.43 and later) takes the name as it is,
+    where `git rev-parse --verify CHERRY_PICK_HEAD` also resolves a branch
+    or a tag of that name, which no operation leaves. A git before 2.43
+    refuses the option and is answered no: such a git has no reftable
+    either, so the file `_in_progress` asks first is the whole answer
+    there."""
+    return _try_git(["show-ref", "--exists", name], sub) is not None
 
 
 def _sequence_in_progress(gitdir: Path) -> tuple[str, str, str] | None:
@@ -718,11 +758,12 @@ def _sequence_in_progress(gitdir: Path) -> tuple[str, str, str] | None:
     return SEQUENCER_TODO, "a cherry-pick", "cherry-pick"
 
 
-def _say_in_progress(label: str, path: str,
-                     row: tuple[str, str, str]) -> None:
+def _say_in_progress(label: str, path: str, row: tuple[str, str, str],
+                     held: str) -> None:
     """A leg with an operation in progress (#704): what is in progress,
-    that nothing moved, and the two commands that end it, the leg's path
-    shell-quoted, then spelled, as `move_back_to_pin` prints its command.
+    where git holds its marker (`held`, #764), that nothing moved, and the
+    two commands that end it, the leg's path shell-quoted, then spelled, as
+    `move_back_to_pin` prints its command.
 
     No move is made from here. A checkout of the tracking branch, even from
     HEAD on it already or detached at the pin, ends a merge, a cherry-pick
@@ -736,7 +777,7 @@ def _say_in_progress(label: str, path: str,
     marker, doing, verb = row
     command = f"git -C {spelled(shlex.quote(path))} {verb}"
     print(f"  {label}: {doing} is in progress in the leg ({marker} is in its "
-          "git dir), so nothing moved")
+          f"{held}), so nothing moved")
     print(f"          finish it (`{command} --continue`) or abort it "
           f"(`{command} --abort`), then run bootstrap again.")
 
@@ -1030,7 +1071,7 @@ def checkout_tracking_branch(root: Path, leg: dict, branch: str) -> None:
                        f"refs/remotes/origin/{branch}"], sub)
     doing = _in_progress(sub)
     if doing is not None:
-        _say_in_progress(where, path, doing)
+        _say_in_progress(where, path, *doing)
         return
 
     if local is None:
@@ -1071,30 +1112,24 @@ def shape_upstream_notice(root: Path) -> None:
     an absent answer is not a finding, and it never fails the bootstrap. A
     clone named there that git cannot be STARTED to read is no absent
     answer but a machine with no git, refused `git-not-runnable` (#694).
+
+    A READ THE SYSTEM REFUSES IS AN ABSENT ANSWER TOO (#764), and so is a
+    `contracts/shape-pin.yaml` that is not a mapping, which `validate-pins.py`
+    has refused by then: each ended `make bootstrap` in a traceback, exit 1.
+    The two reads of the answer itself are `_clone_tip` and `_cached_tip`.
     """
     pin_path = root / "contracts" / "shape-pin.yaml"
-    if not pin_path.is_file():
-        return
     try:
-        pin = load_yaml(pin_path)
-    except Refusal:
+        pin = load_yaml(pin_path) if pin_path.is_file() else None
+    except (OSError, Refusal):
         return
-    pinned = str((pin or {}).get("commit") or "").lower()
+    pinned = str((pin if isinstance(pin, dict) else {}).get("commit")
+                 or "").lower()
     if not COMMIT_40.match(pinned):
         return
-    tip = None
-    upstream = os.environ.get(SHAPE_UPSTREAM_ENV)
-    if upstream and Path(upstream).is_dir():
-        with _git_runs(f"the clone {SHAPE_UPSTREAM_ENV} names, "
-                       f"{spelled(upstream)}, could not be read"):
-            tip = _try_git(["rev-parse", "HEAD"], Path(upstream))
+    tip = _clone_tip(os.environ.get(SHAPE_UPSTREAM_ENV))
     if tip is None:
-        cache = root / SHAPE_UPSTREAM_CACHE
-        if cache.is_file():
-            # `utf-8-sig` takes off the byte-order mark a Windows editor may
-            # start the file with (#855). `strip()` keeps U+FEFF, so a marked
-            # tip read as no commit and the notice said nothing.
-            tip = cache.read_text(encoding="utf-8-sig").strip()
+        tip = _cached_tip(root)
     if not tip or not COMMIT_40.match(tip):
         return
     if tip.lower() == pinned:
@@ -1105,6 +1140,48 @@ def shape_upstream_notice(root: Path) -> None:
           f"is wrong until you have.")
 
 
+def _clone_tip(upstream: str | None) -> str | None:
+    """`HEAD` of the clone `SHAPE_UPSTREAM_PATH` names, or `None` where no
+    clone there can be read (#764).
+
+    A directory this process may not enter is no clone it can read. Python
+    3.12 and 3.13 raise `PermissionError` from `Path.is_dir` for a path
+    under one, and `subprocess` raises it for a `cwd` it cannot enter,
+    naming that directory, which `_git_runs` passes on as not git's. Each
+    ended `make bootstrap` in a traceback, exit 1, where the notice is
+    silent over every other absent answer, and the cache is still asked. A
+    git that cannot be STARTED is still `git-not-runnable` (#694): that is
+    the `Refusal` `_git_runs` raises, which `except OSError` does not catch.
+    """
+    if not upstream:
+        return None
+    try:
+        if not Path(upstream).is_dir():
+            return None
+        with _git_runs(f"the clone {SHAPE_UPSTREAM_ENV} names, "
+                       f"{spelled(upstream)}, could not be read"):
+            return _try_git(["rev-parse", "HEAD"], Path(upstream))
+    except OSError:
+        return None
+
+
+def _cached_tip(root: Path) -> str | None:
+    """What `.shape-upstream-tip` holds, less a byte-order mark it starts
+    with (#855), or `None` where it cannot be read: not there, a file this
+    process may not read, or bytes that are not UTF-8, none of which is a
+    commit (#764). The last two ended `make bootstrap` in a `PermissionError`
+    or a `UnicodeDecodeError` traceback."""
+    cache = root / SHAPE_UPSTREAM_CACHE
+    try:
+        # `utf-8-sig` takes off the byte-order mark a Windows editor may
+        # start the file with (#855). `strip()` keeps U+FEFF, so a marked
+        # tip read as no commit and the notice said nothing.
+        return (cache.read_text(encoding="utf-8-sig").strip()
+                if cache.is_file() else None)
+    except (OSError, UnicodeDecodeError):
+        return None
+
+
 def _register_paths(root: Path, legs: list[dict]) -> list[Path]:
     candidates = [root / REGISTER_RELPATH]
     for leg in legs:
@@ -1312,11 +1389,12 @@ def _checkout_all_tracking_branches(root: Path, sub_legs: list[dict],
 
 
 def _run_neutral_validators(root: Path, manifest: dict | None,
-                            skip: bool) -> list[str]:
+                            skip: bool) -> list[tuple[str, int]]:
     """Step (b): run the neutral validators, then the upstream notice.
 
-    Split from `main` for #132. Returns the labels of any validator that
-    failed, exactly as `main`'s local `failed` list used to accumulate them.
+    Split from `main` for #132. Returns the label and the exit code of each
+    validator that failed, in the order they ran: `_end_of_run` says which
+    were findings and which refusals (#764).
     """
     if skip:
         print("  skipped (--skip-validators)")
@@ -1334,11 +1412,41 @@ def _run_neutral_validators(root: Path, manifest: dict | None,
         print(f"  --- {label} ---")
         code = _run(root, argv_)
         if code != 0:
-            failed.append(f"{label} (exit {code})")
+            failed.append((label, code))
     shape_upstream_notice(root)
     return failed
 
 
+def _end_of_run(failed: list[tuple[str, int]]) -> int:
+    """The last line or two of the run, and its exit code (#764).
+
+    A validator's exit 2 is its REFUSAL, that it could not check, and is said
+    as one: `bootstrap REFUSED: <name> (exit 2)`, exit 2, the code every
+    other refusal here ends with. Any other non-zero exit is a finding, as it
+    was: `bootstrap FAILED: <name> (exit <code>)`, exit 1. Where both
+    happened both lines are said, the findings first, and the run exits 2,
+    since a check that could not run outranks a finding.
+    """
+    # Stdout first: piped, it is block-buffered and stderr is not, so a log of
+    # both would show the lines below before step (c)'s, as `main`'s
+    # `git-not-runnable` path takes care that it does not.
+    sys.stdout.flush()
+    found = [f"{label} (exit {code})" for label, code in failed
+             if code != VALIDATOR_REFUSED]
+    refused = [f"{label} (exit {code})" for label, code in failed
+               if code == VALIDATOR_REFUSED]
+    if found:
+        print("bootstrap FAILED: " + ", ".join(found), file=sys.stderr)
+    if refused:
+        print("bootstrap REFUSED: " + ", ".join(refused), file=sys.stderr)
+    if refused:
+        return 2
+    if found:
+        return 1
+    print("bootstrap ok")
+    return 0
+
+
 def main(argv: list[str] | None = None) -> int:
     escape_unencodable_stdout()
     parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
@@ -1380,11 +1488,7 @@ def main(argv: list[str] | None = None) -> int:
     read_authority(root, legs)
 
     print()
-    if failed:
-        print("bootstrap FAILED: " + ", ".join(failed), file=sys.stderr)
-        return 1
-    print("bootstrap ok")
-    return 0
+    return _end_of_run(failed)
 
 
 if __name__ == "__main__":

tests/test_bootstrap_placement_texts.py

@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: Apache-2.0
+"""`docs/cli.md` says what the assembly root's bootstrap places with the
+condition #704 holds it to (#764, C3).
+
+THE DEFECT. The flag-level reference's row for
+`templates/assembly-root/scripts/bootstrap.py` said it "places each leg on
+its tracking branch AT the pinned commit", with no condition. Since #704
+(Brett Heap's R1 and R2, 2026-10-16T13:07:34Z) it places a leg there only
+where that loses no commit, and `AGENTS-shape.md`, which every project
+carries, says so in R2's words. The row is generated from
+`scripts/render-cli-reference.py`, so `tests/test_cli_reference.py` held the
+two in step and neither to the rule.
+
+THE RULE NOW. The row carries R2's condition, word for word as the pinned
+`AGENTS-shape.md` says it. The other pinned sentences, which say bootstrap
+places a leg there "or says why not" (the Makefile's help, the family
+holder's `AGENTS-shape.md` and README, and `AGENTS-shape.md`'s own table of
+five commands), are true as they stand, so R2 left them and so does this.
+"""
+
+from __future__ import annotations
+
+import re
+
+from conftest import REPO
+
+#: R2's condition, as the pinned `AGENTS-shape.md` words it.
+CONDITION = ("only where that loses no commit, and otherwise leaves its "
+             "checkout where it is, saying why")
+
+
+def one_line(path) -> str:
+    """The file's text with every run of white space made one space, so a
+    sentence wrapped across lines reads as one."""
+    return re.sub(r"\s+", " ", path.read_text(encoding="utf-8"))
+
+
+def test_the_pinned_agent_file_still_says_r2s_condition():
+    """The words this file holds the reference to are R2's own."""
+    agents = one_line(REPO / "templates" / "assembly-root" /
+                      "AGENTS-shape.md")
+    assert ("`make bootstrap` places a leg on its tracking branch AT the "
+            f"pin {CONDITION}.") in agents
+
+
+def test_the_reference_row_for_the_assembly_bootstrap_says_the_condition():
+    row = next(line for line in (REPO / "docs" / "cli.md").read_text(
+        encoding="utf-8").splitlines()
+        if line.startswith("| [`templates/assembly-root/scripts/"
+                           "bootstrap.py`]"))
+    assert (f"places each leg on its tracking branch AT the pinned commit "
+            f"{CONDITION}. What `make bootstrap` runs. |") in row, row