adopt-project.py: execute records in project.yaml which leg the plan gave each submodule a leg keeps, and shape-doctor.py holds the submodule to that row, so one in the other leg is MISPLACED and a real adoption's placement reads ok (#189)

AGENTS.md

@@ -152,6 +152,11 @@ order, and step 4 is the one you must not skip.
    and the split are pushed and verified: finish with the command it prints
    and never re-run into fresh legs, which abandons that pair.
 
+`execute` also records the plan's answers about the source's submodules in
+the split's `project.yaml`, under `submodules:` — per submodule a leg keeps,
+its `path`, `leg`, `resolution:` and `source_commit`. It confers nothing,
+nothing rewrites it, and the doctor holds each submodule to it.
+
 `execute` opens a pull request, because these organisations are pull-request
 only. Never suggest a direct push to the default branch, and never `--admin`.
 
@@ -242,7 +247,10 @@ which writes that file and nothing else — see 4.
    `review_required: true` is a question the policy is ASKING, not a defect
    found: answering it on their behalf puts a fact in their tree that nobody
    decided. The plan is `kind: placement-plan` and is not an adoption plan;
-   never hand one to `adopt-project.py`, which refuses it by name.
+   never hand one to `adopt-project.py`, which refuses it by name. A
+   submodule is judged by the adoption's `submodules:` record, not a rule
+   (`rule: null`, `confidence: recorded`), and is a `note` where no row names
+   it. One moved on purpose stays MISPLACED: its row is the human's to change.
 5. **`NOT A SHAPE ROOT` is a question for the human, not a task.** There is
    no `project.yaml` and no `family.yaml`. The two ways in are
    `adopt-project.py plan` (a repository that already exists, converted in

adopt-project.py

@@ -46,7 +46,10 @@
            `xf-project-<id>` topic on all three (skipped for local remotes),
            and then VERIFIES by blob sha that every source path landed in
            exactly one place, and that the root's only submodules are the
-           two leg mounts, each at the commit pushed to its leg.
+           two leg mounts, each at the commit pushed to its leg. The split's
+           `project.yaml` records the leg each source submodule kept in a
+           leg was answered into, under `submodules:`, for `shape-doctor.py`
+           to check it against.
 
 THE PLAN IS AN ARTIFACT A HUMAN OR AN AI EDITS. That is why it is YAML with
 reasons in it rather than a pipe between two processes: the classifier is
@@ -4901,6 +4904,100 @@ def _where_it_landed(assembly: Path, written: str) -> str | None:
         if landed.is_relative_to(root) else None
 
 
+#: The key `execute` appends to the assembly's `project.yaml` (#189). The
+#: doctor reads it by this name through the module it loads, and the
+#: project's own `validate-manifest.py`, which imports nothing of this tool's,
+#: spells it once more.
+SUBMODULE_RECORD = "submodules"
+
+#: The comment above that block, in the manifest itself, because the
+#: manifest is read by people who never saw the plan it came from.
+SUBMODULE_RECORD_PREAMBLE = (
+    "",
+    "# THE SOURCE'S OWN SUBMODULES, and the leg each one was ANSWERED into",
+    "# when this repository was adopted (#189). Written once, by",
+    "# `adopt-project.py execute`, from the adoption plan: one row per",
+    "# submodule the split kept in a leg, its path inside that leg, the",
+    "# `resolution:` the plan gave the entry that covered it, and the source",
+    "# commit the plan was answered at. `shape-doctor.py` checks each",
+    "# submodule a leg holds against it. Like every field here, it confers",
+    "# nothing.",
+    f"{SUBMODULE_RECORD}:",
+)
+
+
+def submodule_record(entries: list, tree: list, commit: str) -> str:
+    """The `submodules:` block `execute` appends to `project.yaml`, or ""
+    when the split keeps no submodule of the source's in a leg (#189).
+
+    WHY THE ANSWER IS WRITTEN DOWN. Which leg holds a source submodule is a
+    human's answer under #166 -- `plan` asks it of every entry that covers a
+    gitlink, whatever rule its name matches -- and the plan that carries the
+    answer is not committed anywhere. Without a record, the assembly holds
+    `.cursor` in the code leg and nothing else, and `shape-doctor.py` cannot
+    tell that answer from a submodule added by hand, so the most it can say
+    is a note. With one, a submodule in the leg recorded is placed and one in
+    the other leg is MISPLACED, with the manifest as the evidence.
+
+    ONE ROW PER GITLINK KEPT IN A LEG, from `_answered_legs`, the reading
+    `check` holds the plan to: by the time `execute` calls this,
+    `_refuse_what_check_finds` has refused a gitlink no entry covers, one
+    covered twice, one left unanswered and one answered `root`. A gitlink
+    answered `drop` is in no leg, so nothing can be checked against it, and
+    one added at that path later is a gitlink no record covers. The path is
+    the gitlink's path in the source, which `git filter-repo` keeps as its
+    path in the leg. `resolution:` is written when the plan's entry has one;
+    a directory entry's answer covers every gitlink under it, so each row
+    carries that entry's line.
+
+    A SOURCE WITH NO SUBMODULE GETS "" and so the `project.yaml` -- and the
+    split -- it always got, byte for byte.
+
+    EVERY PATH AND REASON IS DOUBLE-QUOTED, by `repo_shape.double_quoted`,
+    the writer of the escapes the reader resolves (#199). Always quoted,
+    unlike `y`: a submodule may sit at `null`, `true` or `0001`, which a
+    plain scalar reads back as something that is not a path. And a character
+    `str.isprintable` refuses is spelled by its code point: NEL and U+2028
+    end a line for the reader, a no-break or zero-width space is another name
+    than the one it looks like, and the surrogate a byte that is not UTF-8 is
+    kept as cannot be written by `write_lf` at all, where `\\udce9` reads
+    back as that surrogate. So every row reads back as the name the doctor
+    holds the submodule at to it, and the manifest is written after both
+    legs exist. A name with a control character or a byte that is not UTF-8
+    in it is one the doctor reads no further: it reports it as a name no
+    report can carry.
+    """
+    gitlinks = [path for path, mode, _, _ in tree if mode == GITLINK_MODE]
+    kept = _kept_in_a_leg(_answered_legs(entries, gitlinks), gitlinks)
+    if not kept:
+        return ""
+    resolutions = {str(e.get("path")): _resolution_text(e.get("resolution"))
+                   for e in entries}
+    lines = list(SUBMODULE_RECORD_PREAMBLE)
+    for path, leg, entry in kept:
+        lines.append(f"  - path: {double_quoted(path)}")
+        lines.append(f"    leg: {leg}")
+        if resolutions.get(entry):
+            lines.append(
+                f"    resolution: {double_quoted(resolutions[entry])}")
+        lines.append(f'    source_commit: "{commit}"')
+    return "\n".join(lines) + "\n"
+
+
+def _resolution_text(resolution) -> str | None:
+    """A plan entry's `resolution:` as the text the record carries, or None.
+
+    A scalar is written as the plan's reader returned it. A blank one is no
+    reason, and a mapping or a list under the key is not one line of text,
+    so neither is written: the record's `resolution:` is optional, and a row
+    without one still says where the submodule was answered into.
+    """
+    if resolution is None or isinstance(resolution, (dict, list)):
+        return None
+    text = str(resolution)
+    return text if text.strip() else None
+
+
 def _commit_the_split(plan: Plan, source: Source, assembly: Path,
                       names: dict, urls: dict, values: dict, work_root: Path,
                       paths_for: dict, seeded: list, spec_path: str,
@@ -4960,9 +5057,17 @@ def _commit_the_split(plan: Plan, source: Source, assembly: Path,
                         spec_path, code_path)
 
         step = "committing the split"
+        # The plan's submodule answers outlive the plan only here (#189).
+        # An empty record appends nothing, so a source with no submodule
+        # gets the manifest it always got.
+        append = {"Makefile": ADOPT_MAKEFILE_BLOCK}
+        record = submodule_record(plan.entries, source.tree(),
+                                  source.commit)
+        if record:
+            append["project.yaml"] = record
         materialized = materialize_assembly_root(
             SHAPE_ROOT, assembly, values, collision_dir=COLLISION_DIR,
-            append={"Makefile": ADOPT_MAKEFILE_BLOCK})
+            append=append)
         for intended, actual in materialized.collisions:
             print(f"  beside  {actual} (the source already has {intended}; "
                   "nothing was overwritten)")

scripts/repo_shape.py

@@ -21,6 +21,8 @@
   - `file_sha256` — the per-file digest used by the shape pin's `files:` block,
     mirroring `neutral-product-pin`'s per-file `sha256` rows.
   - `NamingPolicy` — the classifier over `contracts/repository-naming.yaml`.
+  - `path_spelling_fault` — whether a path is spelled the one way Git
+    records it, asked of a leg's mount path and of a recorded submodule's.
   - `same_repository` and the remote-url arithmetic under it — the ONE
     definition of whether two remote spellings name one repository, which
     `shape-doctor.py` reports by and `scripts/siblings.py` refuses by.
@@ -348,10 +350,13 @@ def double_quoted(text: str) -> str:
     """`text` as a double-quoted scalar `_scalar` reads back as `text`.
 
     The subset's one writer quotes with this: `adopt-project.py`'s `y`, for
-    the adoption plan and for `shape-doctor.py`'s placement plan. It escapes
-    a backslash and a double quote, which would bend or end the quoting;
-    `\\n`, `\\r` and `\\t` by name; and every other character `str.isprintable`
-    refuses by its code point. PART OF THAT CLASS THE READER CANNOT TAKE
+    the adoption plan and for `shape-doctor.py`'s placement plan, and its
+    `submodule_record`, for the `submodules:` record `execute` appends to
+    `project.yaml`, which the doctor holds a submodule to only where a row
+    reads back as its path (#189). It escapes a backslash and a double
+    quote, which would bend or end the quoting; `\\n`, `\\r` and `\\t` by
+    name; and every other character `str.isprintable` refuses by its code
+    point. PART OF THAT CLASS THE READER CANNOT TAKE
     RAW. It reads a file line by line with `str.splitlines`, which ends a
     line at `\\n` and `\\r` -- the macOS folder icon is a file named
     `Icon\\r` -- and also at `\\x0b`, `\\x0c`, `\\x1c`-`\\x1e`, `\\x85`,
@@ -1935,6 +1940,55 @@ def checked_value(what: str, value, pattern: re.Pattern = SAFE_ARG_RE) -> str:
     return text
 
 
+# ---------------------------------------------------------------------------
+# A path spelled the one way Git records it (#179, #189)
+# ---------------------------------------------------------------------------
+#
+# Git records a path in exactly one spelling: relative to the top of its
+# repository, `/` between segments, and no segment empty, `.` or `..`.
+# `git update-index` refuses `./x`, `x/`, `a//b`, `a/./b`, `a/../b`, `/x` and
+# `.` alike as `Invalid path`, so a record that names a path in any other
+# spelling names one no tree holds, and nothing is ever found at it.
+#
+# TWO RULES ASK IT. The leg-path rule in `shape_materialize.py` asks it of a
+# leg's mount path (#179), and a project's own `validate-manifest.py` asks it
+# of each path `project.yaml`'s `submodules:` records (#189) -- which is why
+# it lives HERE: a project carries this file and not `shape_materialize.py`.
+# Each says the answer in its own sentence, about a leg or about a submodule.
+#
+# A BACKSLASH AND A DRIVE LETTER ARE NOT ASKED. On POSIX each is an ordinary
+# character of a name -- Git records `vendor/a\b` and `C:/x` there, refusing
+# them only on Windows -- and an adoption records a source submodule at
+# either. They are faults only in a path a PERSON types for commands that
+# Windows also runs, which is the leg-path rule's own question.
+
+#: The ways a path can fail to be spelled as Git records it, in the order
+#: `path_spelling_fault` asks: empty, absolute, climbing out with `..`,
+#: naming the top itself, and any other empty or `.` segment.
+PATH_EMPTY = "empty"
+PATH_ABSOLUTE = "absolute"
+PATH_CLIMBS = "climbs"
+PATH_ITSELF = "itself"
+PATH_NOT_CANONICAL = "not-canonical"
+
+
+def path_spelling_fault(text: str) -> str | None:
+    """Why `text` is not one path spelled as Git records it, as one of the
+    `PATH_*` codes above, or None when it is."""
+    if not text:
+        return PATH_EMPTY
+    if text.startswith("/"):
+        return PATH_ABSOLUTE
+    segments = text.split("/")
+    if ".." in segments:
+        return PATH_CLIMBS
+    if all(segment in ("", ".") for segment in segments):
+        return PATH_ITSELF
+    if "" in segments or "." in segments:
+        return PATH_NOT_CANONICAL
+    return None
+
+
 def accepts_role(found, role: str) -> bool:
     """Is `found` an acceptable classification for a leg DECLARED as `role`?
 

scripts/shape_materialize.py

@@ -43,8 +43,9 @@
 
 sys.path.insert(0, str(Path(__file__).resolve().parent))
 from repo_shape import (  # noqa: E402
-    CHAIN_RECORD_FIELD, SAFE_ARG_RE, NamingPolicy, Refusal, checked_value,
-    double_quoted, file_sha256,
+    CHAIN_RECORD_FIELD, PATH_CLIMBS, PATH_ITSELF, PATH_NOT_CANONICAL,
+    SAFE_ARG_RE, NamingPolicy, Refusal, checked_value, double_quoted,
+    file_sha256, path_spelling_fault,
 )
 # The reader's own parse and its own comment rule, for "The two lines a pin
 # is moved by" below. `_strip_comment` is that module's private name and is
@@ -2069,6 +2070,14 @@ def copy_out(src: Path, dst: Path) -> None:
 # asks (see "Commit trailers" above for the same argument). This module
 # is in no copy list and both tools already import it.
 #
+# EXCEPT THE SEGMENTS, which a project's copy DOES ask (#189): its
+# `validate-manifest.py` holds each path `project.yaml`'s `submodules:`
+# records to the spelling Git records a path in, so
+# `repo_shape.path_spelling_fault` answers that for both rules, and
+# `_spelling_reason` below adds only what a TYPED path can get wrong -- a
+# drive letter and a backslash, which on POSIX are characters of a name Git
+# records -- and says every answer about a leg.
+#
 #   CANONICAL = a relative POSIX path of one or more segments, each non-empty
 #   and neither `.` nor `..`: no leading `/` or `./` (nor a drive letter), no
 #   trailing `/`, no `//`, no backslash. The two legs also take DIFFERENT
@@ -2132,8 +2141,28 @@ def copy_out(src: Path, dst: Path) -> None:
 )
 
 
+#: What each way a path's SEGMENTS can be misspelled means for a leg's mount
+#: path. `path_spelling_fault` asks the segments; the empty and absolute
+#: answers it also gives are `_spelling_reason`'s own, asked first, beside
+#: the drive letter and the backslash only a typed path can get wrong.
+LEG_SEGMENT_REASONS = {
+    PATH_CLIMBS: ("it has a `..` segment, so it names a different place than "
+                  "it spells and can climb out of the assembly root"),
+    PATH_ITSELF: ("it names the assembly root itself, and a leg is mounted "
+                  "INSIDE the root, not at it"),
+    PATH_NOT_CANONICAL: (
+        "Git records a mount without its `.` segments (a leading `./`), a "
+        "trailing `/` or a doubled `/`, so the mount and its `.gitmodules` "
+        "entry would name two different paths"),
+}
+
+
 def _spelling_reason(text: str) -> str | None:
-    """Why `text` is not a CANONICAL spelling of a leg path, or None."""
+    """Why `text` is not a CANONICAL spelling of a leg path, or None.
+
+    The segments are `repo_shape.path_spelling_fault`'s, the rule a
+    project's own validator also holds a recorded submodule path to (#189).
+    """
     if not text:
         return "it is empty"
     if text.startswith("/") or WINDOWS_ABSOLUTE_RE.match(text):
@@ -2142,18 +2171,7 @@ def _spelling_reason(text: str) -> str | None:
                 "its path is relative to it")
     if "\\" in text:
         return "it has a backslash, and a path here is written with `/`"
-    segments = text.split("/")
-    if ".." in segments:
-        return ("it has a `..` segment, so it names a different place than it "
-                "spells and can climb out of the assembly root")
-    if all(segment in ("", ".") for segment in segments):
-        return ("it names the assembly root itself, and a leg is mounted "
-                "INSIDE the root, not at it")
-    if "" in segments or "." in segments:
-        return ("Git records a mount without its `.` segments (a leading "
-                "`./`), a trailing `/` or a doubled `/`, so the mount and its "
-                "`.gitmodules` entry would name two different paths")
-    return None
+    return LEG_SEGMENT_REASONS.get(path_spelling_fault(text))
 
 
 def _unmountable_reason(text: str) -> str | None: