Document subprocess coverage measurement

articles/_release-notes/v0.13.x.md

@@ -6,6 +6,32 @@ list_title: Versions 0.13.x
 
 # Release Notes - Versions 0.13.x
 
+## Version 0.13.22
+
+### New Features
+* Coverage is now collected from the Python subprocesses a covered task spawns,
+  and from anything those spawn in turn. PyBuilder plants a startup hook into
+  the site directories of the VEnvs it builds and hands the active
+  *coverage.py* configuration over through the environment, so a process that
+  starts from one of those VEnvs measures itself from its own startup. Code
+  that only ever runs in a subprocess - a build backend invoked by `pip`, a
+  program under test started with `subprocess` - is no longer reported as
+  uncovered. Nested PyBuilder builds started from an integration test plant the
+  hook into their own VEnvs, so the whole process tree is measured. Controlled
+  by the new `coverage_subprocesses` property, on by default, and settable per
+  task as `ut_coverage_subprocesses` / `it_coverage_subprocesses`. With
+  `--no-venvs` nothing is planted into the Python you started the build with;
+  subprocess measurement rides on *coverage.py*'s own startup hook there.
+* Measurement in the unit test tool process now starts at interpreter startup
+  rather than once the tool has loaded, so imports and module level code are
+  measured too.
+
+### Changed Features
+* The `coverage` plugin now requires *coverage.py* 7.13 or newer. The
+  configuration hand-off needs `COVERAGE_PROCESS_CONFIG` and
+  `CoverageConfig.serialize`, and `--no-venvs` builds additionally need
+  *coverage.py*'s own startup hook, which is what 7.13 adds.
+
 ## Version 0.13.21
 
 ### New Features

documentation/manual.md

@@ -529,12 +529,45 @@ override:
    - A `CoverageTool` is registered with the reactor
    - The Python environment for the covered task is overridden to inject a coverage
      shim script into the subprocess
+   - A startup hook is planted into the site directories of the VEnvs PyBuilder built,
+     and the active coverage configuration is handed to the task through the
+     environment, so that further subprocesses measure themselves
    - The covered task (e.g. `run_unit_tests`) is re-executed with coverage active
    - Coverage data is collected from the subprocess via `coverage.combine()`
 
 3. **After execution**: Coverage data from all covered tasks is aggregated, thresholds
    are checked, and reports are generated.
 
+### Subprocess Measurement
+
+Code that only ever runs in a subprocess - a build backend `pip` invokes, a program
+under test started with `subprocess`, a worker started with `os.system()` - would
+otherwise report as entirely uncovered: the VEnvs PyBuilder builds into have no
+`coverage` installed, and nothing running at interpreter startup to switch it on.
+
+The startup hook PyBuilder plants closes that gap. Any Python process started from one
+of those VEnvs picks the hook up from its own `site` processing, imports `coverage`
+from PyBuilder's plugin VEnv without leaving it on the path, and starts measuring
+before the process runs a line of its own code. That also means imports and module
+level code are measured, not just what runs once the test harness has control. The
+collected data is written alongside the covered task's own data and combined into it.
+
+The hand-off is inherited, so this applies to the whole process tree, including the
+VEnvs created by a nested PyBuilder build running inside an integration test.
+
+With `--no-venvs` there are no VEnvs of PyBuilder's making, and PyBuilder plants nothing
+into the Python you started the build with. Subprocesses are still measured there, by
+*coverage.py*'s own startup hook, which that Python has because `--no-venvs` installs
+`coverage` into it.
+
+Set `coverage_subprocesses` to `False` to turn this off, either for the whole project
+or for a single task (`ut_coverage_subprocesses`, `it_coverage_subprocesses`).
+Measuring every process in the tree is not free, and that includes processes you may
+not care about, such as `pip`.
+
+This requires *coverage.py* 7.13 or newer, which is where its own startup hook - the
+only thing that can act on the hand-off under `--no-venvs` - arrived.
+
 ### Thresholds
 
 Coverage enforces three independent threshold types:

documentation/plugins.md

@@ -142,6 +142,13 @@ Use the ```python.coverage``` module to activate coverage.
     <td>Concurrency libraries to measure. See <a href="https://coverage.readthedocs.io/en/latest/config.html#run-concurrency">coverage.py concurrency</a> for supported values.</td>
   </tr>
 
+  <tr>
+    <td>coverage_subprocesses</td>
+    <td>bool</td>
+    <td>True</td>
+    <td>Measure Python subprocesses spawned by a covered task, and anything those spawn in turn. Can also be set per task, for example <code>ut_coverage_subprocesses</code> for unit tests or <code>it_coverage_subprocesses</code> for integration tests.</td>
+  </tr>
+
   <tr>
     <td>coverage_source_path</td>
     <td>string</td>
@@ -152,6 +159,18 @@ Use the ```python.coverage``` module to activate coverage.
 
 Please note that properties `coverage_allow_non_imported_modules`, `coverage_reset_modules`, `coverage_reload_modules` and `coverage_fork` are deprecated and will not be used.
 
+#### Measuring subprocesses
+
+Code that only ever runs in a subprocess used to report as entirely uncovered: a program under test invoked through `subprocess`, a build backend that `pip` calls, a worker started with `os.system()`. The VEnvs *PyBuilder* builds into have no `coverage` installed and nothing running at interpreter startup to switch it on, so those processes ran unmeasured.
+
+With `coverage_subprocesses` enabled, *PyBuilder* plants a startup hook into the site directories of the VEnvs it built, and hands the active *coverage.py* configuration to the covered task through the environment. Every Python process started from one of those VEnvs then measures itself from its own startup, which also means imports and module level code are measured rather than only what runs once the test harness has control. The results are combined into the task's coverage data as usual.
+
+This applies recursively, so a nested *PyBuilder* build started by an integration test plants the hook into the VEnvs it creates as well, and the whole process tree is measured.
+
+With `--no-venvs` there are no VEnvs of *PyBuilder*'s making and nothing is planted into the Python you started the build with. Subprocesses are still measured there, by *coverage.py*'s own startup hook, which that Python has because `--no-venvs` installs `coverage` into it.
+
+Set the property to `False` to turn it off, either globally or for a single task. Note that measurement is not free: every Python process in the tree pays for it, including ones you may not care about such as `pip`.
+
 ### SonarQube integration
 
 Using the plugin `python.sonarqube` will add the task `run_sonar_analysis` to your project. This task has a run-time dependency to the `sonar-runner` program and will submit your project to [sonarqube](http://www.sonarqube.org/) for analysis.