Proposed changes from migration experiments: a stacked series of 23 pull requests #35

Summary. any2bazel-lab, EngFlow's experiment harness for this skill, has made 86 unattended migration
runs on 14 open-source CMake projects, from a few dozen to about 5,000 translation units. In each run an agent
(Claude, in Claude Code, given one fixed prompt) migrates a project to Bazel. The lab then judges the result
independently:

• its own reference build;
• the skill's differ, re-run on the lab's own captures of both builds' action graphs;
• the project's tests and runtime output;
• generated files;
• a fresh build.

The evidence comes in two parts, on the same project commits, container images, model and client version:

• Before: 70 runs, with the skill as published at 210b25a (the two earliest at an older commit) or without
  it.
  • Four small projects (graphicsgems, jansson, lensfun, pono) were run at least three times in each arm.
  • Ten larger ones, of about 500 to 5,000 translation units, were run once in each arm (CMake and
    mysql-server twice): bitcoin, blender, CMake's own sources, freecad, gdal, mysql-server, openttd, qgis,
    xbmc (Kodi) and yugabyte-db.
• After: 16 runs, all with the skill carrying these changes, on eight of the same projects: graphicsgems,
  lensfun, pono, openttd, CMake, xbmc, qgis and yugabyte-db. The first ten runs used the changes' first
  version. Six re-runs then used the revised version, after the first ten showed three proposals needed work.

The changes are on a branch rebased onto c1ca1b4, the current main. That is eight commits after
210b25a, mainly a reorganization of SKILL.md into docs/ and the aquery param-file change (#29).

This issue proposes 23 changes, as a stack of pull requests on c1ca1b4:

• U items (U01–U17) are for review: fixes and additions with direct evidence, from runs or from the
  lab's stored data.
• D items (D1–D6) are for discussion, per CONTRIBUTING.md. Each is a judgment about how an agent
  should behave, or a trade-off for interactive use, and those are the maintainers' to make.

Each PR's description carries its own evidence. The lab's records are in EngFlow/any2bazel-lab, readable
inside EngFlow.

What the runs found (before)

• The verdict did not see some differences that matter (U01, U04, U05).

  • Canonicalization discarded optimization and debug levels. 8 of 9 pono migrations, with or without the
    skill, built at -O2 against an -O3 reference, and 0 of 7 lensfun migrations kept -g. The differ
    reported no difference: one pono run had 0 errors at -O2 against -O3.
  • The NDEBUG that -c opt adds was invisible.
  • A source compiled in several variants gave a verdict that depended on PYTHONHASHSEED.
  • Uppercase CONFIGURE_FILE was not extracted.

  U05 reports these differences; whether any of them should block converged is left to you.

• Converged did not mean done (U06, U17). SKILL.md says "Done when both stages converge" (line 59 at
  c1ca1b4), with no build required. Three of the ten larger migrations with the skill (gdal, xbmc,
  mysql-server) failed bazel build //..., with kodi and mysqld among the missing targets. Configure-time
  generated files were not compared:

  • a lensfun config.h differed in 3 of 7 converged runs;
  • in one CMake run, three configure headers that the agent generated and committed differed from the
    reference.

  Of the 79 migrations whose trees the lab keeps, three had Bazel files that the project's own .gitignore
  hid from a commit.

• Noise that agents suppressed by hand (U07, U08). Every lensfun, jansson and pono run with the skill
  suppressed the same compile-only flags that CMake repeats on link lines. Every graphicsgems run needed 30
  to 32 target_map entries, none of them a rename. A suppression made by hand can also hide a real
  difference.

• Prerequisites and commands that fail unattended (U11, D6). SKILL.md lists jq as required, and says to
  stop and ask if it is missing, but no script uses it. The aquery command it says to "Always" use fails
  analysis with the toolchain rules_cc autodetects on Linux.

What the after runs showed

graphicsgems and openttd were run once; lensfun, pono, xbmc and qgis twice; CMake and yugabyte-db three
times.

• pono kept -O3; lensfun and openttd kept -g.
• The three CMake headers came out byte-identical in all three CMake runs.
• xbmc (Kodi), one of the three failed builds, built every target by vendoring FFmpeg, SWIG and four smaller
  libraries.
• A later xbmc run also ran its tests under bazel test: 1,686 of the 1,689 reference-passing tests passed.
  That run's build, though, relied on an FFmpeg its agent had installed on the host instead of vendoring
  it. U17 now rules that out, in a sentence written after the runs.
• In the six re-runs, no agent suppressed a link flag by hand, and none installed jq. In the first ten, two
  still suppressed link flags that U08's first version had missed, and two downloaded jq.

One thing our own first version made worse. D4's first wording told agents to "run builds and test
suites that may take minutes in the background and poll them". Agents did, with fixed sleeps of a guessed
length and with pgrep -f waits that match their own command line. Across the first ten after-runs, 232
minutes went to commands still waiting after the work they waited for had ended. One run waited 74 minutes
on a 16-second configure. With D4's revised text, the six re-runs spent 6 minutes that way, against 153 for
the same six projects with the first wording. Waiting after the work has ended is a behavior, not a cost
estimate, so one run per project is enough to show it.

The series

The PRs are stacked because several touch the same lines of diff.py and SKILL.md. Each is based on the
one before, starting from c1ca1b4, so that each diff stays small. We have not mapped which ones depend on
which in substance. If one is declined, we will rebase the rest.

Kinds:

• fix: the verdict was wrong;
• report: the differ's output gains information without changing a verdict;
• guidance: SKILL.md or docs/REFERENCE-cmake.md text;
• discussion: as above.

U01–U10 change scripts/ with tests; the rest change SKILL.md and docs/REFERENCE-cmake.md only.

What was tested, and how.

• U01 and U09 were on the branch the runs used, but no run exercised them; they were checked offline on
  stored data.
• U11, U17's last sentence and D4's last wording change were written after the runs.
• At every PR's commit the full test suite passes under pytest (130 tests on main, 196 at the top of the
  stack). CI has not run yet, since no PR is open.
• The series' final tree is identical to the branch the re-runs tested, plus those three later changes.

Questions for the maintainers

1. The discussion items.

   • D3 narrows SKILL.md's list of unsupported cases that trigger a stop, to build-time generation with
     no Bazel mapping.
   • D2: an agent with nobody to confirm with records its scope findings, and the decision it took, in
     any2bazel.json under scope_findings, then proceeds. The stop for unsupported cases still holds.
   • D1: the agent sets aside a project's agent instructions where they conflict with the migration,
     and says so.
   • D4: the agent runs long commands in the foreground when the tool's timeout allows. Otherwise it
     backgrounds them with a wait that ends when the command does.
   • D5 is a warning whose wording and placement are a judgment call.
   • D6 drops jq from the prerequisites, which interactive users may still want.

   Are these the behaviors you want?

2. U08's built-in list of compile-only flags is a design choice you may prefer to own, or to make
   configurable.

3. Shape. Is a stack the form you want, or would you rather have the U items as independent PRs, each
   rebased onto main as the earlier ones land?

Limits of the evidence

At the larger sizes there is one run per project per skill version, and variation from run to run is large.
The two first-round CMake runs, at the same commit and with the same skill version, differed 2.4× in
notional cost (tokens priced at API list price). A changed behavior shows up in one run: a wait, a
suppression, a test that now runs. An effect on total time or cost does not, so this issue claims none.

The pull requests

• U01: Match configure_file case-insensitively in the trace extractor #36
• U02: Validate that bazel_args holds flags, not a command #37
• U03: Decide target_map renames in one place, and list renamed exclusions #38
• U04: Compare every variant of a multi-variant source, deterministically #39
• U05: Report each side's optimization and debug level, Bazel-only flags and defines, and NDEBUG #40
• U06: Say what convergence does not cover: configure-time files and shared-library link lines #41
• U07: Align executables by unambiguous label basename #42
• U08: Report compile-only flags on CMake link lines as a warning #43
• U09: Compare link-interface flags from CMake's libraries fragments #44
• U10: Count link-only targets' external libraries in the link-closure check #45
• U11: Say what to do when the toolchain refuses to disable linker param files #46
• U12: Pin the reference configuration, record its command, and require parity for the default invocation #47
• U13: Tell the agent to map the build type to flags, not to a compilation mode #48
• U14: Name the three configure-step strategies and their costs #49
• U15: Say that a git-derived version stamp is lost unless stamped #50
• U16: Say where the parity artifacts belong #51
• U17: Make a reproducible clean build part of done: vendoring, declared tests, no host installs, nothing gitignored #52
• D1: Treat a project's own agent instructions as information (for discussion) #53
• D2: Say what a non-interactive agent should do at the scope check (for discussion) #54
• D3: Keep "stop" only for build-time generation with no Bazel mapping (for discussion) #55
• D4: Run long commands so that waiting ends with the command (for discussion) #56
• D5: Warn that Bazel's defines are transitive (for discussion) #57
• D6: Say that jq is optional (for discussion) #58