shape-doctor.py: run_validator starts a root's OWN validator under the doctor's interpreter with site processing on, so a .pth in the operator's user site or in a virtualenv's site-packages that names a directory holding repo_shape.py stands in for the copy the root lost, and the manifest row says ok ... passes for a gate that fails in CI on ModuleNotFoundError: the false pass #238 removed and #306 closes for PYTHONPATH alone #342

Lane: openreposhape-4 (openRepoShape-4)

Found by this lane's adversarial review of #306 (PR #311, head 9d30bdb; its out-of-scope item O1). That review showed a user-site .pth naming the standard's scripts/ turning the manifest row of a root with no scripts/repo_shape.py into ok, and suggested weighing -s / PYTHONNOUSERSITE=1. Re-verified on main c7c02fa and at #311's landing head b30420f. Main has not touched shape-doctor.py, the template validators or scripts/repo_shape.py since #311's base ede15ce, and git merge-tree of the two is clean, so b30420f's doctor is byte for byte what main will run once #311 lands, and the finding survives that landing. Each run had PYTHONPATH unset, a project scaffolded against local bare remotes, Python 3.12.3, and a gh shim (the machine row's preflight calls it; it exits 97).

New here:

• A .pth in a virtualenv's own site-packages does the same thing when the doctor is run by that virtualenv's python.
• -s does not stop that route. Of the interpreter flags, only -S stops both, measured on the root's own validator run directly.

Not claimed:

• Windows, macOS, and other Pythons.
• The family row (validate-family.py goes through the same run_validator): read, not run.
• usercustomize and the system's own site-packages: the same mechanism, not run.
• That anything in the standard writes such a .pth: nothing does. It is the operator's or their environment's.

What is wrong. #306's fix gives a root's own validator a child with NO PYTHONPATH, so that "the row reports the project's own GATE as CI runs it". That is run_validator's docstring (shape-doctor.py:964-970 at b30420f); validator_scripts says "NOTHING IS NOTHING" (:907). But PYTHONPATH is one of several ways a directory reaches sys.path. At start-up site also appends every directory named by a .pth file in the user site-packages (or $PYTHONUSERBASE's) and in the interpreter's own site-packages: a virtualenv's, or the system's. The child is subprocess.run([sys.executable, str(script), *args], ...) (:989) with site on, so whatever the doctor's interpreter loads, the root's validator loads too:

# $U: a clean checkout at c7c02fa. The project is scaffolded against local bare remotes; PYTHONPATH is unset throughout.
R=$(mktemp -d) && cd "$R"
python3 "$U/scaffold-project.py" --org testorg --project Atlas --elected-by 'Test Human' \
    --local-remote-dir "$R/remotes" --work-dir "$R/work" > scaffold.log 2>&1
git -c protocol.file.allow=always clone -q --recurse-submodules "$R/remotes/Atlas.git" Atlas
(cd Atlas && python3 scripts/bootstrap.py > ../bootstrap.log 2>&1)
mv Atlas/scripts/repo_shape.py .                             # a root that lost its own copy
UB="$R/userbase"; SITE=$(PYTHONUSERBASE="$UB" python3 -m site --user-site); mkdir -p "$SITE"
echo "$U/scripts" > "$SITE/standard.pth"                     # one line: the standard's scripts/
doctor() { (cd / && env -u PYTHONPATH "$@" python3 "$U/shape-doctor.py" --root "$R/Atlas" 2>&1 | grep -E '^ *(ok|FINDING) +(manifest|pins) |^(COMPLIANT|DRIFTED|INVALID)'); }
doctor                                                       # nothing on the path but the interpreter's own
#     FINDING  manifest         the project's own scripts/validate-manifest.py reports a finding (exit 1): ModuleNotFoundError: No module named 'repo_shape'
#     FINDING  pins             the project's own scripts/validate-pins.py reports a finding (exit 1): ModuleNotFoundError: No module named 'repo_shape'
#   DRIFTED (1 copy-missing)   (exit 1)
doctor PYTHONUSERBASE="$UB"                                  # the operator's user site holds the .pth
#     ok       manifest         the project's own scripts/validate-manifest.py passes
#     FINDING  pins             the project's own scripts/validate-pins.py reports a finding (exit 1): FINDING shape-copy-missing: contracts/shape-pin.yaml: scripts/repo_shape.py is pinned but absent
#   DRIFTED (1 copy-missing)   (exit 1)
(cd Atlas && env -u PYTHONPATH python3 scripts/validate-manifest.py --root . 2>&1 | tail -n 1; echo "exit ${PIPESTATUS[0]}")   # the gate as CI runs it
#   ModuleNotFoundError: No module named 'repo_shape'
#   exit 1

This is #306's own picture with a different route in. The verdict is DRIFTED (1 copy-missing) both times, because shape currency ranks first and names the missing file. What flips is the manifest row: a FINDING for everybody else, and ok ... passes for this operator, over a gate that fails in CI. The pins row is a finding either way, with another message.

The doctor at b30420f, run with --json beside main's over a root built the same way, gives the same two rows in each case. So does the doctor started by a virtualenv's python whose site-packages holds the same .pth (python3 -m venv --without-pip, the .pth in its purelib), and that case is unchanged under PYTHONNOUSERSITE=1.

The root's own validator run directly shows which interpreter flags stop which route:

Why. The env block decides PYTHONPATH and nothing else, and .pth files are read by site when the interpreter starts, whatever the environment says. A validator does sys.path.insert(0, <its own directory>) before from repo_shape import, and a .pth directory is appended after the standard library. So a .pth can never SHADOW a repo_shape.py the root still has. It can only fill the gap when the root has lost one, which is exactly the case #238's row exists to report. A CI runner's fresh interpreter has no such .pth.

(An operator who pastes the row's NEXT gets ok too, under the same .pth. #306's decision 2 already says the row is about the gate, not the paste.)

What it should do. The own-copy row says what the project's gate says, whatever the operator's site-packages hold. The decisions are the author's:

1. How.
   • -S runs with no site at all: no .pth from any site-packages, no user site, no sitecustomize or usercustomize. It still honours PYTHONPATH, so the template run keeps its <shape>/scripts.
   • -s / PYTHONNOUSERSITE=1 covers the user site only. A virtualenv's or the system's .pth still stands in (the table).
   • -I covers the user site only and also drops PYTHONPATH. That is usable for the own copy, which gets none anyway, and never for the template run (the shape-doctor.py: after #238 the doctor runs a root's OWN validator with the operator's PYTHONPATH still in the child's environment, so for an operator who exports one that reaches a repo_shape.py the manifest row of a root that lost its own scripts/repo_shape.py says ok ... passes again, the false pass #238 removed, where the project's own gate in CI fails on ModuleNotFoundError: No module named 'repo_shape' #306 review's point).
   • Or the docstrings say instead that the row is the gate "under this interpreter's site-packages".
2. Which children. Either the own copy only, the one run a .pth can decide, or every validator child for one rule: the template, the naming script, and the machine row's setup-project.py --preflight. The validators import the standard library and repo_shape only (shape-doctor.py: after #238 the doctor runs a root's OWN validator with the operator's PYTHONPATH still in the child's environment, so for an operator who exports one that reaches a repo_shape.py the manifest row of a root that lost its own scripts/repo_shape.py says ok ... passes again, the false pass #238 removed, where the project's own gate in CI fails on ModuleNotFoundError: No module named 'repo_shape' #306's review listed them), so -S costs them nothing. Check the preflight, and what -S does to Debian's dist-packages, before including it.
3. The printed NEXT. shape-doctor.py validator_row falls back to the standard's TEMPLATE validator when a root carries no copy of its own, and prints that path as the NEXT command without the PYTHONPATH it runs it under, so pasting the manifest, pins or family row's NEXT dies with ModuleNotFoundError: No module named 'repo_shape' and exit 1, the code a validator uses for a finding #238 and shape-doctor.py: after #238 the doctor runs a root's OWN validator with the operator's PYTHONPATH still in the child's environment, so for an operator who exports one that reaches a repo_shape.py the manifest row of a root that lost its own scripts/repo_shape.py says ok ... passes again, the false pass #238 removed, where the project's own gate in CI fails on ModuleNotFoundError: No module named 'repo_shape' #306 keep "the run and the line apply PYTHONPATH under one condition". Decide whether validator_command prints the same flag (python3 -S scripts/validate-manifest.py ...) so that a paste agrees with CI, or whether run_validator's "WHICH AGREEMENT THE ROW KEEPS" paragraph names site-packages beside the reader's PYTHONPATH. tests/test_shape_doctor.py pins the NEXT lines, and the PowerShell spelling (shape-doctor.py validator_row falls back to the standard's TEMPLATE validator when a root carries no copy of its own, and prints that path as the NEXT command without the PYTHONPATH it runs it under, so pasting the manifest, pins or family row's NEXT dies with ModuleNotFoundError: No module named 'repo_shape' and exit 1, the code a validator uses for a finding #238) would carry the flag too.
4. Platforms. -S on the python.org and setup-python builds CI uses on macOS and Windows keeps the standard library on sys.path. CI's three jobs are the check.

How to test. Next to #311's test_an_exported_pythonpath_does_not_stand_in_for_a_lost_repo_shape (tests/test_shape_doctor.py:5567 at b30420f), over the same root (a valid manifest, its own repo_shape.py removed):

• Run the doctor with PYTHONUSERBASE=<tmp_path>/userbase, whose user site holds a .pth naming the standard's scripts/. Assert the manifest row is a FINDING quoting ModuleNotFoundError; today it is ok.
• If decision 1 covers it, a virtualenv variant: python -m venv --without-pip in tmp_path, the .pth in its purelib, and the doctor started by that interpreter.

Mutants: drop the flag and both fail; use -s where the decision is -S and the virtualenv row fails.

Verified at main c7c02fa (shape-doctor.py sha256 4e3bfd9c…) and at #311's head b30420f (f1731ff4…, identical to that file after #311 merges into main).

Related:

• shape-doctor.py: after #238 the doctor runs a root's OWN validator with the operator's PYTHONPATH still in the child's environment, so for an operator who exports one that reaches a repo_shape.py the manifest row of a root that lost its own scripts/repo_shape.py says ok ... passes again, the false pass #238 removed, where the project's own gate in CI fails on ModuleNotFoundError: No module named 'repo_shape' #306 (PR shape-doctor.py: a validator's child gets exactly the PYTHONPATH its row's NEXT prints -- the standard's scripts/ alone for the template copy, none at all otherwise -- so an operator whose PYTHONPATH reaches a repo_shape.py is no longer told a root that lost its own scripts/repo_shape.py passes its gate, and the template run no longer keeps the operator's directories behind the standard's (#306) #311: PYTHONPATH taken out of the child; this is that review's O1).
• shape-doctor.py validator_row falls back to the standard's TEMPLATE validator when a root carries no copy of its own, and prints that path as the NEXT command without the PYTHONPATH it runs it under, so pasting the manifest, pins or family row's NEXT dies with ModuleNotFoundError: No module named 'repo_shape' and exit 1, the code a validator uses for a finding #238 (the own copy runs as the project runs it).
• shape-doctor.py: validator_row reads a validator's exit 1 as "reports a finding" whatever caused it, so a validator that CRASHED (a : in the standard's path splits the PYTHONPATH the template fallback runs under, and validate-manifest.py dies on import repo_shape) is a FINDING row quoting ModuleNotFoundError, and a root whose manifest is valid goes from COMPLIANT, SHAPE BEHIND to INVALID (manifest) #278 (validator_row reads any exit 1 as a finding).
• shape-doctor.py: for a root whose project.yaml does not load AND whose scripts/repo_shape.py is gone, the manifest row's NEXT promises its validator "refuses it the same way until it loads", but pasted it dies on ModuleNotFoundError: No module named 'repo_shape', exit 1, so the person is sent to a command that cannot say what the row quoted #298 (the manifest row's NEXT for a root whose manifest does not load and whose repo_shape.py is gone).