[brettheap]

Lane: openreposhape-5 (openRepoShape-5)
Claim: #260 (comment)

What was wrong (#260). render replaced the placeholders one key at a time, each over the whole text. The value an earlier key put in was then searched by every later key. --elected-by and --reference in both tools, the scaffold's --name, and family.py init's --name and --created-by are free text, so each could hold another placeholder's token:

• A token for a key that came later was expanded. adopt-project.py --elected-by '{{SPEC_PATH}}' recorded elected_by: "spec" in project.yaml and the README, and said adoption verified.
• A token for a key that came earlier ({{PROJECT}}), or one no key names ({{NOPE}}), stayed in the output. render then refused it as template-unsubstituted, with a remediation that blamed the tool. By then execute had pushed both legs, the scaffold had created all three repositories, and family.py init had created the holder's bare repository.

Reproduced on b30595c with the issue's own script, and the same for family.py init --created-by.

The change (scripts/shape_materialize.py).

• render reads the template once. One regular-expression substitution looks each placeholder up in the table, so a value goes in where its placeholder stood and is never searched again. In a .yaml/.yml template the quoted form "{{KEY}}" is matched first and spelled by double_quoted, exactly as repo_shape.py: a quote opens a quoted scalar only where one begins and must close there, so k: [it's, a] reads two items, k: it's fine # note drops its comment, and an unclosed "abc or a mid-key [ is refused, not misread (#249, #250) #284 made it (YAML_PLACEHOLDER_RE, group 1).
• The refusal asks the template, not the output: it names the placeholders the template holds that the table has no value for, and it is raised before anything is substituted. That is the defect its remediation describes, so the remediation is now true. The output now holds the person's text, so it is no longer searched. Message and code are unchanged: REFUSED template-unsubstituted: <source>: no value for ['{{NOPE}}'].
• PLACEHOLDER_RE now takes digits, \{\{[A-Z0-9_]+\}\}, a superset of the old [A-Z_]+. The four digest placeholders ({{SPEC_TREE_SHA256}}, {{CODE_TREE_SHA256}}, {{SHAPE_TREE_SHA256}}, {{PIN_TREE_SHA256}}) were substituted only because the loop replaced every key by name. The single pass replaces only what the pattern finds, so it needs the digits. A digest placeholder that a table lacked used to be left in the output with nothing said; it is now refused like any other.

Nothing downstream renders these files again, so a {{ a person typed is never expanded later, and no tool needs to refuse {{ in these values. On this head:

• git grep -n -E '\brender\(' -- '*.py' ':!tests' lists scripts/shape_materialize.py alone: copy_tree and the three calls in _materialize. (scripts/render-cli-reference.py has its own unrelated render.)
• copy_tree, materialize_assembly_root and materialize_family_root are called only from adopt-project.py execute (:2549, :3101), scaffold-project.py (:1013, :1054) and family.py init (:712). Every one of those calls runs while the tool is creating the repositories.
• update-shape.py imports no renderer and no TEMPLATED list. In project.yaml / family.yaml it rewrites only the shape: block's commit: and tree_sha256: lines (rewrite_manifest, :869). The README's trailing Shape: line is a span rewrite of the commit.
• bump-leg.py rewrites only a leg's two pin lines.
• shape-doctor.py imports only escape_what_stdout_cannot_encode and lane_trailer_argument from the materializer. It reports a leg's AGENTS.md/README.md as rendered and compares nothing.
• Neither bootstrap.py nor any shipped validator holds {{: git grep -n -F '{{' -- 'templates/*.py' finds only ^{{commit}} in two f-strings.
• shape_advisory.py's own PLACEHOLDER_RE is asked only of single-repository.yaml (field_problems, called only from read_record). That is a record a person writes by hand, never project.yaml.

Every caller, and what it relies on.

• adopt-project.py: _template_values (ORG, REFERENCE, ELECTED_BY, and so on) is rendered into seeded legs and the assembly root. It relies on every key a template names being in the table (it is: the four leg commit/digest keys are "" until the legs exist). Its free text is now written as typed.
• scaffold-project.py: the values table, plus _seed_legs' four leg values and --pin's PIN_* values for neutral pins. Same reliance; the probe after this list names the keys.
• scripts/family.py: _template_values. FAMILY is a key no template names, and that is harmless.
• setup-project.py renders nothing. It forwards --elected-by and --name to the scaffold, and its own NAME_RE already refuses {.
• update-shape.py and shape-doctor.py render nothing (above).
• Tests: test_adopt_leg_path_collisions.py and test_scaffold_leg_path_collisions.py build a table from PLACEHOLDER_RE.findall over templates/assembly-root/. They now include the digest keys and still pass. test_shape_advisory.py renders leg AGENTS.md and the record template, which have no digest keys. test_scaffold_ignored_shape_files.py calls copy_tree. tests/test_template_quoted_values.py built its tables with its own [A-Z_]+, so its tables lacked the digest keys and its own "SHA256" in key branch never ran. It now uses [A-Z0-9_]+, and repo_shape.py: a quote opens a quoted scalar only where one begins and must close there, so k: [it's, a] reads two items, k: it's fine # note drops its comment, and an unclosed "abc or a mid-key [ is refused, not misread (#249, #250) #284's behaviour is otherwise held unchanged.

A probe (ast over each tool's dict literals, against every placeholder under templates/) finds that every capitals key of the scaffold's and adopt's tables is named by a template, except ORG, which only family-root and workspace-root name. Every key of family's table is named by a template, except FAMILY.

Byte-for-byte for ordinary values. Every text file under templates/ (49) was rendered with b30595c's render and with this head's, over two tables: plain values, and values holding ", ', \, #, [x], {y}, U+3000 and a tab. That is 0 differences in 2 x 49 files. test_an_ordinary_value_renders_as_a_plain_substitution keeps this as a test for the 19 templates that hold a placeholder.

Tests (tests/test_template_values_single_pass.py, 133).

• render: each key's token (every key of the three tools' tables and every key a template names, 49 keys) and {{NOPE}}, bare and between double quotes, as every free-text value. Each reads back unchanged from project.yaml and family.yaml. Every README and AGENTS.md holds it exactly where it holds an ordinary value. Neither of two keys reads the other's value, in either order. A placeholder the template lacks is still refused, naming only the template's missing tokens and never a value's; this includes a digit key. ${{ github.x }} and an f-string's {{commit}} are left alone.
• End to end, into local bare remotes, with a gh on PATH that records any call (none is made). What was pushed is read back with git show, so no checkout's line endings are compared:
  • scaffold-project.py --name/--elected-by/--reference, each holding every key's token (bare and quoted), and each as {{NOPE}}. The run exits 0. The pushed project.yaml, README and both legs' AGENTS.md hold the values as typed, and the project's own validate-manifest.py passes.
  • adopt-project.py plan / check / execute with --elected-by and --reference the same way. All three pass, with adoption verified. The split commit's project.yaml and README hold the values as typed. This test is skipped at its decorator without git filter-repo.
  • family.py init --created-by the same way. family.yaml holds the value, and validate-family.py passes. (The family's --name is held by the unit tests only. Its validator requires the holder's name to be one CamelCase token, so a free-text name is refused there whatever render does. See the last item under Left out.)

On b30595c this file is 113 failed, 20 passed. The 20 are the byte-for-byte guard (19), which describes base behaviour by design, and the sanity test that the templates are found.

Mutation probes, each a revert of one part of the fix: the sequential loop with the template check (98 failed); single pass with the output check (109 failed); PLACEHOLDER_RE without digits (69 failed, the scaffold e2e among them, because the pins keep {{SHAPE_TREE_SHA256}}); no quoted branch (94 failed, #284's tests among them); no template check (4 failed).

Targeted runs on this head: tests/test_template_values_single_pass.py tests/test_template_quoted_values.py tests/test_scaffold_e2e.py tests/test_scaffold_pin_and_reuse.py tests/test_adopt_plan.py tests/test_adopt_execute_refusals.py tests/test_adopt_e2e.py tests/test_family.py tests/test_repo_hygiene.py tests/test_cli_reference.py tests/test_adopt_leg_path_collisions.py tests/test_scaffold_leg_path_collisions.py tests/test_shape_advisory.py tests/test_scaffold_ignored_shape_files.py, with git-filter-repo 2.47.0 on PATH: 792 passed, 1 skipped (test_adopt_plan.py:393, SHAPE_MEDXEHR_CLONE unset). tests/test_shape_doctor.py: 208 passed.

Composition (git merge-tree --write-tree against the current tips; each merged tree ran the sibling's own test files plus this PR's two):

Branch	Tip	Merge	Merged-tree tests
origin/main	c0ed615	clean	680 passed
fix/scaffold-manifest-values (lane 5's #258/#261, PR #314)	7defac7	clean	370 passed
fix/adopt-template-values-utf8 (lane 2's #270)	edfa7c6	clean	278 passed
fix/leg-path-yaml-scalars (lane 2's #266)	b2b42f8	clean	409 passed
fix/adopt-check-shape-paths (lane 5's #262/#263, PR #313)	65a9262	clean	320 passed

• scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so --id null, --tracking-branch true and a blank --elected-by are refused (#258, #261) #314 calls render in manifest_misreads and manifest_invalid with the complete table, so the template-side refusal never fires there. It now finds placeholders with this PLACEHOLDER_RE.
• scaffold-project.py: a leg path the project's own YAML reader would read back as another type (null, ~, true, False, 2026, 1.5) is refused before anything is created, under --dry-run alike, through the one definition adopt-project.py asks, so --spec-path 2026 no longer creates three repositories whose first bootstrap fails manifest-leg-path, and adopt-project.py execute no longer says adoption verified before that bootstrap fails (#225) #266 has merged main, so the import-line conflict an earlier tip had is gone.
• Lane 1's adopt-project.py plan writes null, true and other YAML keywords unquoted, so plan --spec-path null silently mounts the leg at the default spec #181 (fix/adopt-plan-quote-yaml-keywords, e337f44): a clean merge, one failing test, and the fix belongs to whichever lands second. adopt-project.py plan writes null, true and other YAML keywords unquoted, so plan --spec-path null silently mounts the leg at the default spec #181 alone passes (841). On the merged tree, 1 of 974 fails: test_adopt_records_each_free_text_value_as_typed[every-key], refused adopt-manifest-misread at plan. adopt-project.py plan writes null, true and other YAML keywords unquoted, so plan --spec-path null silently mounts the leg at the default spec #181's gate _written (e337f44 adopt-project.py:1668-1669) spells the value with double_quoted (lane 1's 4bf2cc4), then runs .replace(token, value), which re-reads the value it just put in. That is the per-key re-scan this PR removes. So with both, the gate refuses values execute now writes as typed: --elected-by 'x{{ELECTED_BY}}' (the gate claims "xx{{ELECTED_BY}}"), '"{{ELECTED_BY}}"', and --reference 'see {{REFERENCE}}' (it claims "see see {{REFERENCE}}"). On this PR alone all three pass plan and are written as typed. Neither branch is wrong against main, where render really would expand them. Whichever lands second makes _written one pass for its key: one re.sub over "(\{\{KEY\}\})"|\{\{KEY\}\}, or render on the line, as scaffold-project.py and adopt-project.py ask the project's YAML reader and validate-manifest.py about the manifest's values before anything exists, so --id null, --tracking-branch true and a blank --elected-by are refused (#258, #261) #314's _misread_rows does. Tokens of other keys ({{SPEC_PATH}} {{NOPE}}) pass on every tree.

Left out.

• No template placeholder is quoted (templates/assembly-root/project.yaml and the two leg pins write eight text placeholders bare (id, elected_on, visibility, tracking_branch, the two leg paths and submodule_paths), so a checked value YAML reads as another type (007, 2026, true, null) is misread by the project or refused by the tool; quoting them reads every one back exactly and lets numeric branch names adopt #259 waits on Brett), and no free-text value is refused for holding {{, since nothing renders it again.
• family.py init --name takes a display name (its help says so) that the holder's own validate-family.py refuses. On b30595c, --name 'Contoso Group' exits 0 and pushes the holder, and validate-family.py then exits 1 with naming-unclassified: 'Contoso Group' matches no family in the naming policy. This is pre-existing and not touched here; it is filed as family.py init exits 0 and pushes a holder its own validate-family.py refuses for --id null, --created-by ' ', --created-on 20261011 or --name 'Contoso Group', and puts --tracking-branch true on a branch its manifest reads back as True #307, with family.py init's other values.

Review. The first reviewer stalled at its first checkpoint and was replaced. The second review of 552763d found the code sound and asked only that this body's composition be corrected (above). Its #181 paragraph described an earlier #181 tip, and every other tip had moved.

What the review checked:

• adopt-project.py and scaffold-project.py substitute the template placeholders one after another, so a free-text value holding another placeholder's token is expanded (--elected-by '{{SPEC_PATH}}' is recorded as elected_by: "spec") or, after the legs are pushed, refused as template-unsubstituted with "a defect in the substitution table of the tool you ran" #260's rows reproduced on b30595c and pass on the head, for adopt, scaffold and family.py init.
• All 49 templates render byte-identically between base and head over 25 tables of awkward values, in both key orders, and 69 files pushed by the real tools are byte-identical.
• A 40,000-case fuzz of render against an independent single-pass reference found 0 mismatches.
• 322 hostile values round-trip through project.yaml and family.yaml under both this repository's reader and PyYAML.
• Every table holds every digest key its templates name, so the widened refusal cannot fire on a run that used to succeed.

Nits kept:

• Three mutants survived the first head. Fix round 1 (321e511, tests only, with main c0ed615 merged as 7d6c66c) gives each a test that fails on it:

  • a .yml source gets the quoted spelling as a .yaml one does;
  • the template-unsubstituted list is the sorted set of what the template lacks, each token once;
  • an empty string in the table is a value that renders as empty text and is never refused.

  The seven targeted files pass on the merged tree, 541 tests.

• The gh stub proves nothing on Windows, where no test calls gh.

Closes #260

🤖 Generated with Claude Code

Summary by Sourcery

Make template materialization substitute placeholders once and validate missing template values before writing repositories.

Bug Fixes:

• Ensure template placeholders are substituted in a single pass so free-text values containing placeholder-like tokens are preserved exactly as provided.
• Report only placeholders missing from the template's substitution table, before materialization begins, preventing false refusals after repositories have been pushed.
• Recognize digit-containing placeholders such as digest keys and refuse them when their values are unavailable.

Enhancements:

• Preserve quoted placeholder handling consistently for YAML and YML templates while retaining existing value escaping behavior.

Tests:

• Add comprehensive unit and end-to-end coverage for single-pass substitution, missing placeholders, quoted YAML values, empty values, digest placeholders, and preservation of literal brace expressions.

[brettheap]

(lane openRepoShape-5, 2026-10-12T02:24Z) @codex review

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 21 hours and 5 minutes by commenting @sourcery-ai review. Upgrade to get a review now.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[sourcery-ai]

Reviewer's Guide

The PR fixes placeholder substitution so values containing placeholder-like text are written literally, while genuinely missing placeholders are rejected from the source template before repository creation or pushing. It also recognizes digest placeholders containing digits, retains YAML quoted-value handling, and adds broad unit and end-to-end regression coverage across scaffold, adoption, and family initialization.

Sequence diagram for safe placeholder substitution

sequenceDiagram
    participant Tool
    participant Renderer
    participant Template
    participant Values
    participant Repository
    Tool->>Renderer: render(text, values, source)
    Renderer->>Template: Read placeholders once
    Renderer->>Values: Look up each template placeholder
    alt Missing placeholder in template
        Renderer-->>Tool: Refusal: template-unsubstituted
    else All placeholders have values
        Renderer->>Renderer: Substitute in one pass
        Note over Renderer: Inserted value text is never searched again
        Renderer-->>Tool: Rendered text
        Tool->>Repository: Create and push repositories
    end

Loading

Flow diagram for single-pass template rendering

flowchart LR
    T[Template source] --> M{Missing placeholders?}
    M -- yes --> R[Refuse before substitution]
    M -- no --> S[Single regex substitution]
    S --> O[Rendered output]
    V[Values table] --> S
    O --> P[Repository creation and push]

Loading

File-Level Changes

Change	Details	Files
Reworked placeholder rendering to perform template-driven, single-pass substitution with accurate pre-substitution validation.	Expanded placeholder recognition to include digits for digest keys. Detect missing placeholders from the original template and refuse before substitution. Prevent substituted free-text values from being rescanned for placeholder tokens. Preserved YAML quoted-placeholder escaping while matching quoted forms first. Left workflow and f-string brace patterns outside the placeholder grammar unchanged.	scripts/shape_materialize.py
Added comprehensive regression coverage for typed free-text substitution across renderer and materialization workflows.	Covered cross-key, unknown-token, quoted-value, digit-key, and ordinary-value behavior. Verified scaffold, adopt-project, and family initialization outputs from pushed commits. Updated quoted-value test patterns to include digit-containing placeholders. Checked validators and ensured no GitHub CLI calls occur in local end-to-end runs.	tests/test_template_values_single_pass.py tests/test_template_quoted_values.py

Assessment against linked issues

Issue	Objective	Addressed	Explanation
#260	Change template rendering to perform placeholder substitution in a single pass, so values containing placeholder-like tokens are inserted literally and are not expanded by subsequent substitutions.	✅
#260	Detect missing placeholders from the original template rather than from rendered output, while recognizing all supported placeholder names including digit-containing digest tokens.	✅
#260	Ensure adopt-project.py and scaffold-project.py no longer corrupt or reject free-text values containing placeholder tokens, including refusing genuine missing placeholders before repository creation or pushes.	✅

Possibly linked issues

• adopt-project.py and scaffold-project.py substitute the template placeholders one after another, so a free-text value holding another placeholder's token is expanded (--elected-by '{{SPEC_PATH}}' is recorded as elected_by: "spec") or, after the legs are pushed, refused as template-unsubstituted with "a defect in the substitution table of the tool you ran" #260: The PR directly fixes issue adopt-project.py and scaffold-project.py substitute the template placeholders one after another, so a free-text value holding another placeholder's token is expanded (--elected-by '{{SPEC_PATH}}' is recorded as elected_by: "spec") or, after the legs are pushed, refused as template-unsubstituted with "a defect in the substitution table of the tool you ran" #260 by using single-pass rendering and pre-substitution missing-placeholder validation.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud