Skip to content

fix(tests): stop the runner's own variables leaking into the tests it runs - #2797

Open
earbona23 wants to merge 2 commits into
maester365:mainfrom
earbona23:fix/pester-config-variable-capture
Open

earbona23 wants to merge 2 commits into
maester365:mainfrom
earbona23:fix/pester-config-variable-capture

Conversation

@earbona23

@earbona23 earbona23 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

build-validation (ubuntu-latest) has been failing three tests in Test-MtCaEmergencyAccessExists.Tests.ps1 while macOS and Windows stay green. This is why, and it is not the tests being flaky.

The mechanism

In coverage mode pester.ps1 passes every test file to one Invoke-Pester call, from its own scope:

$config = New-PesterConfiguration
...
$result = Invoke-Pester -Configuration $config

A test that reads a variable it never assigned resolves it up the scope chain, and in that single combined invocation the chain reaches the runner. So $config inside a test is the runner's PesterConfiguration.

Test-MtCaEmergencyAccessExists.Tests.ps1 has five cases that pass $config to the check while meaning "no configuration supplied", so that it auto-detects:

InModuleScope Maester -Parameters @{ C = $config } { Test-MtCaEmergencyAccessExists -EmergencyAccessAccounts $C }

$config is only assigned inside the other It blocks, so in a per-file run it is genuinely $null there. Measured, with a Write-Host added to one of those cases on a throwaway copy:

without coverage:  DIAG config-is-null=True   type=            value=null
with coverage:     DIAG config-is-null=False  type=PesterConfiguration   value={"Run":{"Path":…

The check receives a PesterConfiguration instead of $null, skips auto-detection, and returns $false. Three cases fail. Two more pass for the wrong reason — they expect $false and a PesterConfiguration also produces $false.

Coverage is enabled only on the Linux leg (PESTER_COVERAGE: "${{ runner.os == 'Linux' && 'true' || 'false' }}"), which is exactly where the failures appear.

Reproduced, both large and small

On unmodified main at 02a220b2:

run result
full suite, per-file mode (-TestGeneral:$false) 3,989 tests, 0 failures
full suite, PESTER_COVERAGE=true 8,756 tests, 4 failures
one file, per-file mode, twice 20 passed, 0 failed
one file, PESTER_COVERAGE=true, twice 17 passed, 3 failed, the same three every time

The four in the full coverage run are these three plus the byte-order-mark test, which is #2795. Together the two PRs take that leg to zero. CI's own Linux leg reported 4 of 8,760 on an unrelated PR of mine, which is the same four.

Note what the single-file reproduction rules out: it is not an interaction between test files, and it is not the coverage instrumentation. A direct Invoke-Pester with the same CodeCoverage configuration — profiler or breakpoints, module pre-imported or not — passes 20 of 20. Only going through pester.ps1 fails, because only then is the runner's $config in the scope chain.

The fix

  • pester.ps1: the two locals a test could capture become $pesterConfig and $pesterRun. That is the fix: it protects every test, present and future, without anyone having to know this rule.
  • The five cases now say $null instead of relying on a variable not existing, so the file states its premise.

$result is renamed as cheap insurance rather than a known bug: two files reference $result without assigning it in the same It, and I checked that both pass in either mode today. Drop that half if you prefer the minimum change.

Verified

one file, per-file mode                 20 passed, 0 failed
one file, PESTER_COVERAGE=true          20 passed, 0 failed
rename alone, test file untouched       20 passed, 0 failed   <- the rename is what fixes it
Test-MtCa* in coverage mode            105 passed, 0 failed, JaCoCo report written (1.6 MB)

The third line is the one that matters for review: with only pester.ps1 changed and the test file exactly as it is on main, the three cases pass. The test-side change is clarity, not the repair.

No test asserts this behaviour, and I have not added one. A test that a test file cannot see the runner's variables would have to run pester.ps1 inside Pester, which is more machinery than the one-line rename deserves — but if you want it, the shape is a case that runs pester.ps1 -Coverage over a fixture file that fails when $pesterConfig is visible to it.

Summary by CodeRabbit

  • Tests
    • Updated automated checks for emergency-access policy scenarios, including automatically detected exclusions and combinations of identity types. Expected pass/fail outcomes remain unchanged.
    • The combined test run continues to use the same settings and produce the same test and coverage reports. No user-facing behavior changes are included.

… runs

In coverage mode pester.ps1 passes every test file to one Invoke-Pester call
from its own scope, so a test that reads a variable it never assigned resolves
it up the scope chain and finds the runner's. `$config = New-PesterConfiguration`
was therefore visible to the tests as `$config`.

Test-MtCaEmergencyAccessExists.Tests.ps1 has five cases that pass `$config` to
the check while meaning "no configuration", so that it auto-detects. In coverage
mode they received the PesterConfiguration object instead: three failed and two
passed for the wrong reason. Coverage runs only on the Linux leg of
build-validation, which is why the failures appeared there and nowhere else.

The runner's locals become `$pesterConfig` and `$pesterRun`, and the five cases
say `$null` instead of relying on a variable not existing.
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2db5d42 and eee23cf.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The emergency-access tests now pass $null for -EmergencyAccessAccounts, with their expected results unchanged. The combined Pester run renames its configuration and result variables. Its settings and test-result handling remain unchanged.

Changes: Emergency-access tests

Layer / File(s) Summary
Pester run and test inputs
powershell/tests/pester.ps1, powershell/tests/functions/Test-MtCaEmergencyAccessExists.Tests.ps1
The combined run uses $pesterConfig and $pesterRun. Five emergency-access test calls pass $null for -EmergencyAccessAccounts; their expected results remain unchanged.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Possibly related issues

  • maester365/maester#2775 — The tests pass $null to exercise automatic setting detection, which aligns with the issue’s test objective. This PR does not implement the proposed engine design.

Possibly related PRs

  • maester365/maester#2758 — This PR also changes the EmergencyAccessAccounts parameter flow for Test-MtCaEmergencyAccessExists.

Suggested reviewers: merill, claude

Merge Risk: Minimal · previous review 2db5d

This change only adjusts test runner variable names and test inputs, so it carries no production impact and no remaining merge risk.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly identifies the main fix: preventing runner variables from leaking into tests during execution.
Description check Passed The description is focused and mostly complete. It explains the failure mechanism, affected tests, fix, reproduction steps, and verification results. It does not use every template heading or complete…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is focused and mostly complete. It explains the failure mechanism, affected tests, fix, reproduction steps, and verification results. It does not use every template heading or complete the checklist, but the required technical context is present.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 02a220b and 2db5d42.

Files selected for processing: 2

  • powershell/tests/functions/Test-MtCaEmergencyAccessExists.Tests.ps1
  • powershell/tests/pester.ps1

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@earbona23

Copy link
Copy Markdown
Contributor Author

@merill two test-infrastructure fixes that together take build-validation (ubuntu-latest) back to green. Flagging them because that leg has been red on main and on every PR that merges it, so the failures are not anyone's diff.

#2795 adds a missing byte order mark to Invoke-MtPackGitHubRequest.Tests.ps1. One byte: it is the only tracked .ps1 in the repository without one, and FileEncoding.Tests.ps1 fails on it.

#2797 is the other three. In coverage mode pester.ps1 passes every file to one Invoke-Pester call from its own scope, so a test that reads a variable it never assigned finds the runner's — $config inside a test is the runner's PesterConfiguration. Five cases in Test-MtCaEmergencyAccessExists.Tests.ps1 pass $config meaning "nothing configured, auto-detect"; three failed and two passed for the wrong reason. Coverage runs only on Linux, which is why only that leg saw it.

Measured on unmodified main at 02a220b2: 8,756 tests and 4 failures with PESTER_COVERAGE=true, 3,989 tests and 0 failures in per-file mode — the same four CI reports. CI's Linux leg on #2797 now reports 1 of 8,756: the three are gone, and the one left is the byte order mark that #2795 fixes.

The fix in #2797 is a rename of two locals in pester.ps1, and it works on its own: with the test file exactly as it is on main, the three cases pass. The five $nulls are clarity, not the repair. Nothing else changes, and the coverage report is still produced.

@earbona23

Copy link
Copy Markdown
Contributor Author

Measured what this fixes, since "the runner's variables leak" is hard to size from the diff.

main's baseline on the Linux leg is 4 failures. This branch runs 1, and the three that disappear are exactly the ones the leak was causing:

Agent Identity and Service Principal policies
  It Should only check user-targeted policies when both Agent Identity and user policies exist
No emergency access config defined (auto-detection mode)
  It Should pass when a group is excluded from all policies (auto-detected)
No emergency access config defined (auto-detection mode)
  It Should pass when a user is excluded from all policies (auto-detected)

The remaining one is the byte order mark, which #2795 removes.

Branch build-validation (ubuntu-latest) Run
a PR carrying only main's baseline (#2694) 4 / 8791 tests failed 43052027124
this PR 1 / 8756 tests failed 43231581070
#2795 3 / 8743 tests failed 43231567048

4 = 1 + 3, so the two PRs are complementary and complete.

Two details that make the diagnosis legible:

  • Test-MtCaEmergencyAccessExists.Tests.ps1 passes 20/20 in isolation with Pester 5.7.1, the version the workflow pins. The three failures only appear in the full run, which is the signature of leaked state rather than a defect in those tests.
  • Linux is the only leg with PESTER_COVERAGE=true. That is why Windows and macOS are green and Linux is not, and why this looks platform-specific when it is not.

One warning for anyone reading those logs: the ##[error]Error: Smoke_Error, Bad Parameter … lines are expected noise from the smoke harness exercising its own negative paths. There are 21 of them, and the count is identical in PRs whose build is green. The real summary is the N / M tests failed! line from powershell/tests/pester.ps1.

28 of the 40 open non-draft PRs show that leg red today, on changes that did not cause it.

@earbona23

earbona23 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Measured evidence that this PR and #2795 are complementary and that together they take build-validation (ubuntu-latest) to zero failures. Read from the CI logs of four runs, no local run involved.

run PR failures which tests
43231626375 #2794 (unrelated, Intune) 4 / 8760 the BOM test, plus Should pass when a group is excluded from all policies (auto-detected), Should pass when a user is excluded from all policies (auto-detected) and Should only check user-targeted policies when both Agent Identity …
43231567048 #2795 3 / 8743 the BOM failure is gone; the three auto-detection ones remain
43231581070 #2797 1 / 8756 the three auto-detection failures are gone; only the BOM one remains
43489737792 #2805 (today, PIM alerts) 4 / 8760 the same four

The subtraction is clean: the baseline is four, #2795 removes exactly the one it is about, #2797 removes exactly the three it is about, and no other failure appears in any of the four runs. The BOM failure names the file in its own message — powershell/tests/functions/Invoke-MtPackGitHubRequest.Tests.ps1 — which is the file #2795 adds the mark to (this PR fixes the other three).

Two notes for whoever picks this up:

  • ubuntu-latest is the only leg that fails because it is the only one running with PESTER_COVERAGE, so it is the only one that executes the OS-independent tests. macOS and Windows are green on all four runs.
  • The Smoke_Error and Smoke_Failed lines in those logs are the self-test fixtures and are not failures of the suite; pester.ps1 counts four, and four is what the throw at its line 285 reports.

The totals differ per run (8743, 8756, 8760) because each branch adds its own tests; the failure counts are what matter.

Every open PR against main inherits these four, so the cost of leaving them is paid by every contributor's red check, not only ours.

@earbona23

Copy link
Copy Markdown
Contributor Author

Still REVIEW_REQUIRED with no reviewer assigned. Same measurement I just posted on #2795, repeated here so this PR stands on its own.

Every open PR that actually runs the suite on Ubuntu is red. Of the 45 open non-draft PRs:

state of build-validation (ubuntu-latest) count
red 28
green, but dorny/paths-filter skipped Pester entirely 2
green, Pester really ran — all completed 8–10 Nov 7
no such check recorded 8

The two green ones from today, #2808 and #2809, touch only website/contributors/contributors.yml and their Run Pester tests step is skipped. Since 11 Nov there is not one open PR where Pester ran on Ubuntu and passed; the most recent is #2782 at 2026-11-10T16:15:39Z.

From the job logs:

The point I had not made before: a check that is red on every PR has stopped being a signal. A real regression arriving tomorrow would land as red-on-already-red and be indistinguishable from the four standing failures — most of all for the 20 authors who have no reason to suspect the red is not theirs.

This one is test-infrastructure only: it stops the runner's own variables leaking into the tests it runs. It does not touch product code and is independent of #2795; either can go first.

Happy to rebase, split, or re-measure anything on request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant