Cut the YAML test runner's memory and CPU so country YAML suites need fewer CI runners #615

Problem

The YAML test runner uses too much memory and CPU for country suites to run efficiently in CI. policyengine-us's pull request CI uses 26 jobs and about 646 runner-minutes per run, and 21 of those jobs are YAML shards.

The suite has 34,843 cases in 5,022 files. Memory drives the sharding: each runner runs one batch per subprocess, with one worker. Each subprocess spends about 40 seconds importing and building the tax-benefit system, and more than half a minute tearing it down.

On policyengine-us main with core 3.33.3, one baseline-states shard (6,724 cases) took 2,088 s wall and 1,548 CPU-seconds, with a 10.4 GB peak. One contrib-states shard (215 cases) took 3,401 s and 2,560 CPU-seconds, with a 12.0 GB peak. Both ran on one macOS host.

Causes in the runner

1. Each case's simulation and every reform system the runner built stayed alive for the whole run. Stop the YAML test runner keeping every case's simulation and reform system #569 fixes this.
2. Each instant the model is asked about built a copy of the whole parameter tree, 16 to 33 MB per instant on policyengine-us, on every system. Resolve parameter nodes at instant on first read #570 fixes this.
3. Each distinct combination of dotted parameter inputs (gov.x.y: value in a case's input) cloned the whole tax-benefit system: about 4.5 s and 1.2 GB per combination.
4. Each reform system held two copies of the parameter tree: the baseline clone it was applied to and the reform's own copy.
5. Nothing let several batches share one built system. Each parallel process built and held its own.

Plan

• Stop the YAML test runner keeping every case's simulation and reform system #569: release per-case state and bound the reform-system cache.
• Resolve parameter nodes at instant on first read #570: resolve parameter nodes at instant on first read.
• This issue's PR, stacked on both:
  • dotted inputs copy only the paths to the parameters they set;
  • reform systems are built on the shared reform-free system, so each holds one parameter tree;
  • the cache frees a reform system before building the one that replaces it;
  • --read-only-arrays makes a formula that writes into an array it read fail;
  • policyengine_core.tools.forked_test_runner.run_batches_in_forks runs batches in children forked from one process that has already built the system.
• A core release, then a policyengine-us change that raises the core floor, runs its YAML suites forked, and cuts YAML CI from 21 jobs to 3.

Measured so far

These were measured locally with the three core changes together and policyengine-us running each suite forked, on the same macOS host as the figures above. All cases passed:

In a Linux container capped at 7.5 GB:

• Four baseline-states workers peaked at 6.36 GB together. This was measured before the reform-system change, which mainly helps contrib.
• The whole policyengine-us YAML suite (34,843 cases) passed with and without --read-only-arrays.

Real CI runs are still to come.