You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(tests): stop the runner's own variables leaking into the tests it runs - #2797
build-validation (ubuntu-latest) has been failing three tests in Test-MtCaEmergencyAccessExists.Tests.ps1 while macOS and Windows stay green. This is why, and it is not the tests being flaky.
The mechanism
In coverage mode pester.ps1 passes every test file to oneInvoke-Pester call, from its own scope:
A test that reads a variable it never assigned resolves it up the scope chain, and in that single combined invocation the chain reaches the runner. So $config inside a test is the runner's PesterConfiguration.
Test-MtCaEmergencyAccessExists.Tests.ps1 has five cases that pass $config to the check while meaning "no configuration supplied", so that it auto-detects:
$config is only assigned inside the otherIt blocks, so in a per-file run it is genuinely $null there. Measured, with a Write-Host added to one of those cases on a throwaway copy:
without coverage: DIAG config-is-null=True type= value=null
with coverage: DIAG config-is-null=False type=PesterConfiguration value={"Run":{"Path":…
The check receives a PesterConfiguration instead of $null, skips auto-detection, and returns $false. Three cases fail. Two more pass for the wrong reason — they expect $false and a PesterConfiguration also produces $false.
Coverage is enabled only on the Linux leg (PESTER_COVERAGE: "${{ runner.os == 'Linux' && 'true' || 'false' }}"), which is exactly where the failures appear.
Reproduced, both large and small
On unmodified main at 02a220b2:
run
result
full suite, per-file mode (-TestGeneral:$false)
3,989 tests, 0 failures
full suite, PESTER_COVERAGE=true
8,756 tests, 4 failures
one file, per-file mode, twice
20 passed, 0 failed
one file, PESTER_COVERAGE=true, twice
17 passed, 3 failed, the same three every time
The four in the full coverage run are these three plus the byte-order-mark test, which is #2795. Together the two PRs take that leg to zero. CI's own Linux leg reported 4 of 8,760 on an unrelated PR of mine, which is the same four.
Note what the single-file reproduction rules out: it is not an interaction between test files, and it is not the coverage instrumentation. A direct Invoke-Pester with the same CodeCoverage configuration — profiler or breakpoints, module pre-imported or not — passes 20 of 20. Only going through pester.ps1 fails, because only then is the runner's $config in the scope chain.
The fix
pester.ps1: the two locals a test could capture become $pesterConfig and $pesterRun. That is the fix: it protects every test, present and future, without anyone having to know this rule.
The five cases now say $null instead of relying on a variable not existing, so the file states its premise.
$result is renamed as cheap insurance rather than a known bug: two files reference $result without assigning it in the same It, and I checked that both pass in either mode today. Drop that half if you prefer the minimum change.
Verified
one file, per-file mode 20 passed, 0 failed
one file, PESTER_COVERAGE=true 20 passed, 0 failed
rename alone, test file untouched 20 passed, 0 failed <- the rename is what fixes it
Test-MtCa* in coverage mode 105 passed, 0 failed, JaCoCo report written (1.6 MB)
The third line is the one that matters for review: with only pester.ps1 changed and the test file exactly as it is on main, the three cases pass. The test-side change is clarity, not the repair.
No test asserts this behaviour, and I have not added one. A test that a test file cannot see the runner's variables would have to run pester.ps1 inside Pester, which is more machinery than the one-line rename deserves — but if you want it, the shape is a case that runs pester.ps1 -Coverage over a fixture file that fails when $pesterConfig is visible to it.
Summary by CodeRabbit
Tests
Updated automated checks for emergency-access policy scenarios, including automatically detected exclusions and combinations of identity types. Expected pass/fail outcomes remain unchanged.
The combined test run continues to use the same settings and produce the same test and coverage reports. No user-facing behavior changes are included.
… runs
In coverage mode pester.ps1 passes every test file to one Invoke-Pester call
from its own scope, so a test that reads a variable it never assigned resolves
it up the scope chain and finds the runner's. `$config = New-PesterConfiguration`
was therefore visible to the tests as `$config`.
Test-MtCaEmergencyAccessExists.Tests.ps1 has five cases that pass `$config` to
the check while meaning "no configuration", so that it auto-detects. In coverage
mode they received the PesterConfiguration object instead: three failed and two
passed for the wrong reason. Coverage runs only on the Linux leg of
build-validation, which is why the failures appeared there and nowhere else.
The runner's locals become `$pesterConfig` and `$pesterRun`, and the five cases
say `$null` instead of relying on a variable not existing.
Reviewing files that changed from the base of the PR and between 2db5d42 and eee23cf.
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
Trigger review
No actionable comments were generated in the recent review.
Walkthrough
The emergency-access tests now pass $null for -EmergencyAccessAccounts, with their expected results unchanged. The combined Pester run renames its configuration and result variables. Its settings and test-result handling remain unchanged.
Changes: Emergency-access tests
Layer / File(s)
Summary
Pester run and test inputs powershell/tests/pester.ps1, powershell/tests/functions/Test-MtCaEmergencyAccessExists.Tests.ps1
The combined run uses $pesterConfig and $pesterRun. Five emergency-access test calls pass $null for -EmergencyAccessAccounts; their expected results remain unchanged.
maester365/maester#2775 — The tests pass $null to exercise automatic setting detection, which aligns with the issue’s test objective. This PR does not implement the proposed engine design.
Possibly related PRs
maester365/maester#2758 — This PR also changes the EmergencyAccessAccounts parameter flow for Test-MtCaEmergencyAccessExists.
Suggested reviewers:merill, claude
Merge Risk: Minimal · previous review 2db5d
This change only adjusts test runner variable names and test inputs, so it carries no production impact and no remaining merge risk.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly identifies the main fix: preventing runner variables from leaking into tests during execution.
Description check
The description is focused and mostly complete. It explains the failure mechanism, affected tests, fix, reproduction steps, and verification results. It does not use every template heading or complete…
Docstring Coverage
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Description check
Explanation
The description is focused and mostly complete. It explains the failure mechanism, affected tests, fix, reproduction steps, and verification results. It does not use every template heading or complete the checklist, but the required technical context is present.
Full details: Docstring Coverage
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer TIP This summary will be updated as you push new changes.
@merill two test-infrastructure fixes that together take build-validation (ubuntu-latest) back to green. Flagging them because that leg has been red on main and on every PR that merges it, so the failures are not anyone's diff.
#2795 adds a missing byte order mark to Invoke-MtPackGitHubRequest.Tests.ps1. One byte: it is the only tracked .ps1 in the repository without one, and FileEncoding.Tests.ps1 fails on it.
#2797 is the other three. In coverage mode pester.ps1 passes every file to one Invoke-Pester call from its own scope, so a test that reads a variable it never assigned finds the runner's — $config inside a test is the runner's PesterConfiguration. Five cases in Test-MtCaEmergencyAccessExists.Tests.ps1 pass $config meaning "nothing configured, auto-detect"; three failed and two passed for the wrong reason. Coverage runs only on Linux, which is why only that leg saw it.
Measured on unmodified main at 02a220b2: 8,756 tests and 4 failures with PESTER_COVERAGE=true, 3,989 tests and 0 failures in per-file mode — the same four CI reports. CI's Linux leg on #2797 now reports 1 of 8,756: the three are gone, and the one left is the byte order mark that #2795 fixes.
The fix in #2797 is a rename of two locals in pester.ps1, and it works on its own: with the test file exactly as it is on main, the three cases pass. The five $nulls are clarity, not the repair. Nothing else changes, and the coverage report is still produced.
Measured what this fixes, since "the runner's variables leak" is hard to size from the diff.
main's baseline on the Linux leg is 4 failures. This branch runs 1, and the three that disappear are exactly the ones the leak was causing:
Agent Identity and Service Principal policies
It Should only check user-targeted policies when both Agent Identity and user policies exist
No emergency access config defined (auto-detection mode)
It Should pass when a group is excluded from all policies (auto-detected)
No emergency access config defined (auto-detection mode)
It Should pass when a user is excluded from all policies (auto-detected)
The remaining one is the byte order mark, which #2795 removes.
4 = 1 + 3, so the two PRs are complementary and complete.
Two details that make the diagnosis legible:
Test-MtCaEmergencyAccessExists.Tests.ps1 passes 20/20 in isolation with Pester 5.7.1, the version the workflow pins. The three failures only appear in the full run, which is the signature of leaked state rather than a defect in those tests.
Linux is the only leg with PESTER_COVERAGE=true. That is why Windows and macOS are green and Linux is not, and why this looks platform-specific when it is not.
One warning for anyone reading those logs: the ##[error]Error: Smoke_Error, Bad Parameter … lines are expected noise from the smoke harness exercising its own negative paths. There are 21 of them, and the count is identical in PRs whose build is green. The real summary is the N / M tests failed! line from powershell/tests/pester.ps1.
28 of the 40 open non-draft PRs show that leg red today, on changes that did not cause it.
Measured evidence that this PR and #2795 are complementary and that together they take build-validation (ubuntu-latest) to zero failures. Read from the CI logs of four runs, no local run involved.
the BOM test, plus Should pass when a group is excluded from all policies (auto-detected), Should pass when a user is excluded from all policies (auto-detected) and Should only check user-targeted policies when both Agent Identity …
The subtraction is clean: the baseline is four, #2795 removes exactly the one it is about, #2797 removes exactly the three it is about, and no other failure appears in any of the four runs. The BOM failure names the file in its own message — powershell/tests/functions/Invoke-MtPackGitHubRequest.Tests.ps1 — which is the file #2795 adds the mark to (this PR fixes the other three).
Two notes for whoever picks this up:
ubuntu-latest is the only leg that fails because it is the only one running with PESTER_COVERAGE, so it is the only one that executes the OS-independent tests. macOS and Windows are green on all four runs.
The Smoke_Error and Smoke_Failed lines in those logs are the self-test fixtures and are not failures of the suite; pester.ps1 counts four, and four is what the throw at its line 285 reports.
The totals differ per run (8743, 8756, 8760) because each branch adds its own tests; the failure counts are what matter.
Every open PR against main inherits these four, so the cost of leaving them is paid by every contributor's red check, not only ours.
Still REVIEW_REQUIRED with no reviewer assigned. Same measurement I just posted on #2795, repeated here so this PR stands on its own.
Every open PR that actually runs the suite on Ubuntu is red. Of the 45 open non-draft PRs:
state of build-validation (ubuntu-latest)
count
red
28
green, but dorny/paths-filter skipped Pester entirely
2
green, Pester really ran — all completed 8–10 Nov
7
no such check recorded
8
The two green ones from today, #2808 and #2809, touch only website/contributors/contributors.yml and their Run Pester tests step is skipped. Since 11 Nov there is not one open PR where Pester ran on Ubuntu and passed; the most recent is #2782 at 2026-11-10T16:15:39Z.
The point I had not made before: a check that is red on every PR has stopped being a signal. A real regression arriving tomorrow would land as red-on-already-red and be indistinguishable from the four standing failures — most of all for the 20 authors who have no reason to suspect the red is not theirs.
This one is test-infrastructure only: it stops the runner's own variables leaking into the tests it runs. It does not touch product code and is independent of #2795; either can go first.
Happy to rebase, split, or re-measure anything on request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
build-validation (ubuntu-latest)has been failing three tests inTest-MtCaEmergencyAccessExists.Tests.ps1while macOS and Windows stay green. This is why, and it is not the tests being flaky.The mechanism
In coverage mode
pester.ps1passes every test file to oneInvoke-Pestercall, from its own scope:A test that reads a variable it never assigned resolves it up the scope chain, and in that single combined invocation the chain reaches the runner. So
$configinside a test is the runner'sPesterConfiguration.Test-MtCaEmergencyAccessExists.Tests.ps1has five cases that pass$configto the check while meaning "no configuration supplied", so that it auto-detects:$configis only assigned inside the otherItblocks, so in a per-file run it is genuinely$nullthere. Measured, with aWrite-Hostadded to one of those cases on a throwaway copy:The check receives a
PesterConfigurationinstead of$null, skips auto-detection, and returns$false. Three cases fail. Two more pass for the wrong reason — they expect$falseand aPesterConfigurationalso produces$false.Coverage is enabled only on the Linux leg (
PESTER_COVERAGE: "${{ runner.os == 'Linux' && 'true' || 'false' }}"), which is exactly where the failures appear.Reproduced, both large and small
On unmodified
mainat02a220b2:-TestGeneral:$false)PESTER_COVERAGE=truePESTER_COVERAGE=true, twiceThe four in the full coverage run are these three plus the byte-order-mark test, which is #2795. Together the two PRs take that leg to zero. CI's own Linux leg reported 4 of 8,760 on an unrelated PR of mine, which is the same four.
Note what the single-file reproduction rules out: it is not an interaction between test files, and it is not the coverage instrumentation. A direct
Invoke-Pesterwith the sameCodeCoverageconfiguration — profiler or breakpoints, module pre-imported or not — passes 20 of 20. Only going throughpester.ps1fails, because only then is the runner's$configin the scope chain.The fix
pester.ps1: the two locals a test could capture become$pesterConfigand$pesterRun. That is the fix: it protects every test, present and future, without anyone having to know this rule.$nullinstead of relying on a variable not existing, so the file states its premise.$resultis renamed as cheap insurance rather than a known bug: two files reference$resultwithout assigning it in the sameIt, and I checked that both pass in either mode today. Drop that half if you prefer the minimum change.Verified
The third line is the one that matters for review: with only
pester.ps1changed and the test file exactly as it is onmain, the three cases pass. The test-side change is clarity, not the repair.No test asserts this behaviour, and I have not added one. A test that a test file cannot see the runner's variables would have to run
pester.ps1inside Pester, which is more machinery than the one-line rename deserves — but if you want it, the shape is a case that runspester.ps1 -Coverageover a fixture file that fails when$pesterConfigis visible to it.Summary by CodeRabbit