From 7b033bee11a53cf7c1fabc26de9e754565740d63 Mon Sep 17 00:00:00 2001 From: "Prashant Rane (ALLYIS INC)" Date: Fri, 16 Oct 2026 00:01:01 -0600 Subject: [PATCH 1/2] Add portable exact-run SCIM validation reports Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../README.md | 4 +- .../Export-ScimValidationReport.ps1 | 1397 +++++++++++++++++ .../Orchestrator_Parameters.json | 2 +- .../SCIM-Validation-Test-Overview.md | 26 +- .../SetupLogicApp-Standard-Agent.md | 42 +- .../StandardLogicApp/Test-Authentication.ps1 | 22 + .../Test-ScimValidationReport.ps1 | 1000 ++++++++++++ .../StandardLogicApp/VERSION | 2 +- .../Validation-Reporting-Changes.md | 118 ++ .../StandardLogicApp/scim-onboarding.agent.md | 99 +- 10 files changed, 2686 insertions(+), 26 deletions(-) create mode 100644 Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 create mode 100644 Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 create mode 100644 Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/README.md b/Microsoft.SCIM.LogicAppValidationTemplate/README.md index 3a2bcc97..74223861 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/README.md +++ b/Microsoft.SCIM.LogicAppValidationTemplate/README.md @@ -25,9 +25,11 @@ The authentication readiness policy is **AND across selected methods**: every de For OAuth-only or federation-only runs, set `scimBearerToken` to `""`. The shared token workflow acquires access tokens, and each test suite refreshes its cache before expiry. There is no static-token fallback. Sending an OAuth access token in an HTTP Bearer header does not mean the endpoint supports caller-supplied static tokens. Entra provisioning connectivity is configured and validated separately with the supported method. +At startup the agent creates a unique version/timestamp/run-ID subfolder directly under its starting current directory. Every run, including setup-blocked runs, exports `validation-run-details.json`, a Markdown summary, and optionally a portable archive using [Export-ScimValidationReport.ps1](StandardLogicApp/Export-ScimValidationReport.ps1), with versioned shareable copies. These contain exact-run timing, source/deployed version, deployment provenance, environment/operator context, and sanitized diagnostics with explicit evidence gaps. Tokens and credentials remain protected; unexposed model/runtime settings are not invented. Review reports before sharing because organizational identifiers and test-user data may remain. + See the [Standard setup walkthrough](StandardLogicApp/SetupLogicApp-Standard-Agent.md) and [test overview](StandardLogicApp/SCIM-Validation-Test-Overview.md) for the current behavior. The generic legacy test descriptions below do not override the Standard authentication contract. -The [authentication change guide](StandardLogicApp/Authentication-Changes.md) explains capability selection, token transport, workflow limits, and verification boundaries. Portable reporting is a separate follow-up change. +The [authentication change guide](StandardLogicApp/Authentication-Changes.md) describes the prerequisite authentication change. The [validation-reporting guide](StandardLogicApp/Validation-Reporting-Changes.md) describes this dependent reporting change, evidence scope, provenance, data transformations, and limitations. --- diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 new file mode 100644 index 00000000..1fe969cb --- /dev/null +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 @@ -0,0 +1,1397 @@ +#requires -Version 7.0 +<# +.SYNOPSIS +Exports sanitized, production validation evidence without changing Azure resources. +.DESCRIPTION +Writes validation-run-details.json and validation-report.md in WorkDirectory. Exit +codes: 0 = complete and ready; 1 = complete but not ready; 2 = incomplete or blocked. +Dot-source this file to use the offline helpers without Azure CLI or file writes. +Only explicitly linked child runs are collected; no "latest run" substitution occurs. + +MetadataPath is optional JSON with these optional fields: + environment: tenantId, subscriptionId, resourceGroupName, logicAppName, + managedIdentity, entraApplication, servicePrincipal, provisioningJob, endpoint, + operatorIdentity (supply identifiers, not credential/configuration objects). + supportedAuthenticationMethods: array containing Bearer, OAuth2ClientCredentials, + WorkloadIdentityFederation (authenticationMethods is accepted as an alias). + applicability: object keyed by exact testName, each { applicable: false, reason: ... }. + preflight: { status: Blocked, reason: ..., startTime: ..., endTime: ... }. + provisioningLogPath: path to a supplied provisioning log (relative to WorkDirectory). + agentRun: { startTime, endTime, status } (explicit setup-to-cleanup window, + separate from the Azure Orchestrator window; timestamps may be null). + agentEventsPath: path relative to WorkDirectory, or an absolute path, to a JSON + array of { name, kind, parentName, startTime, endTime, status, details } events. + Details are sanitized; unknown fields are ignored. Missing/open/invalid timings + remain explicit and never receive export-time substitutes. Agent evidence is + optional for historical runs and does not change Azure gallery readiness. + harness: { name, version, model, reasoning, context } (explicit caller-supplied data only). + knownSecrets: optional literal strings for in-memory defense-in-depth redaction. + source: { repositoryUrl, revision, branch, dirty, description } (caller claims, + stored separately from the locally observed agent/source Git provenance). + historicalRun: { version, authenticationPolicy, selectedMethods, description } + (caller-recorded deployed-run claims, never replaced with the current VERSION). + sourceVersion: { value, sha256 } may record the original source VERSION bytes + separately from that run's deployed version and the current reporting source. + Successful content responses, including JSON null, empty strings, false, zero + and empty arrays, are tracked independently of their values. A null downloaded + body requires a recorded successful HTTP status; an absent output property or + failed/unproven download remains unavailable. + After dot-sourcing, Complete-ScimValidationReport -ReportPath + can correct the legacy empty-string Compose classification without any network + requests. It requires collected empty-string inputs, a two-byte output link, + matching classification failures and zero failed content requests. It never + establishes gallery readiness or rewrites the source file. Write its returned + report to a separate WorkDirectory with Write-ScimValidationReport. + deploymentOverlay: { name, revision, sha256, description, + workflowHashes: [ { name, sha256 } ] } (identifiers/hashes only; never contents). + workflowRuns: [ { workflowName, runId, parentWorkflowName, parentRunId, + parentActionName, repetitionName } ] (optional exact IDs supplied by caller, + useful when secureData intentionally hides a Workflow call's run-ID header). + environment values may be strings or identifier-only objects. Recommended objects: + managedIdentity: { type, tenantId, principalId, clientId }; + entraApplication: { id, appId, displayName }; + servicePrincipal: { id, appId, displayName }; + provisioningJob: { id, servicePrincipalId }; + endpoint: { url, displayName, authenticationType }; + operatorIdentity: { id, displayName, userPrincipalName, type }. + scimAuthenticationMethod: Bearer, OAuth2ClientCredentials or + WorkloadIdentityFederation (direct request transport; independent of gallery + supportedAuthenticationMethods). + All fields are optional; omitted harness fields remain null with explicit reasons. + Source/overlay metadata is caller-attributed, never treated as observed deployment + equivalence. Unknown keys are ignored. workflowRuns contains no credentials. + ARM credentials are refreshed using Azure CLI expires_on with a five-minute + margin. An ARM HTTP 401 gets one credential refresh and one retry; final errors + remain evidence failures. Credentials are never sent to signed content hosts. + ResumeReportPath optionally reuses successfully collected sanitized payloads from + terminal actions in a previous report for the same root/subscription/site. Run + headers, action lists and repetition details are still read fresh. Missing/failed + payloads are retried, old evidence failures are not inherited, and current metadata + is applied normally. Reused payloads are labeled; this is not raw-response replay. + The CLI saves sanitized evidence to validation-run-details.checkpoint.json at + workflow boundaries and the first completed action, then at thirty-second + intervals checked after each action and repetition. Matching root/site + checkpoints resume automatically. Batching avoids repeatedly rewriting a large + inventory for fast, skipped repetitions; no collected evidence is truncated. + Complete repetition-list entries are used directly; incomplete entries still + require detail requests. Skipped repetitions have no execution payload to fetch. + Request count, failure count and total/maximum latency by category are recorded + in runtime.evidenceRequests for checkpointed collection. + The default FailureFocused diagnostic scope retains every action/repetition's + metadata, timing and errors. It collects top-level action payloads and failed or + timed-out repetition payloads. Successful repeated HTTP polls and control-flow + repetitions retain metadata only; their payloads are explicitly out of scope. + Semantic-result producers and child workflow calls remain payload-bearing even + when successful. Completeness is assessed within this declared scope, subject + to separately disclosed security exclusions; it never claims every payload. + Checkpoints are incomplete; canonical reports are replaced only after collection. + WorkflowDirectory VERSION value/hash identify current source, not deployed history. +Unknown metadata is ignored. Never pass parameters.json or appsettings as metadata. +Do not place credentials on a command line; prefer a caller-held KnownSecrets array. +Use an existing PowerShell process: & $exporter @arguments -KnownSecrets $secrets. +Do not interpolate secrets into pwsh -Command/-File arguments or write knownSecrets +to MetadataPath. This exporter never serializes the array or the ARM access token. + +Every nested Workflow action and every returned action repetition is inspected for +an exact child run header, independently of the four historical Orchestrator calls. +Secured call headers are security exclusions, not failed payload downloads. Where +available, a separate, explicitly labeled correlation-only child-run inventory is +collected; it does not assert an exact parent-action mapping. Supply workflowRuns +for exact mappings if the service intentionally hides that association. + +The JSON is canonicalized by key and contains no collection timestamp. Evidence +identifiers and UTC timings are preserved. Runtime status is separate from test +semantics. Missing evidence is recorded and prevents readiness. + +ARM API reference: +https://learn.microsoft.com/rest/api/appservice/workflow-run-actions/list +.EXAMPLE +.\Export-ScimValidationReport.ps1 -WorkDirectory . -SubscriptionId $subscription ` + -ResourceGroupName $group -LogicAppName $app -OrchestratorRunId $run -AgentPath .\scim-onboarding.agent.md +#> +[CmdletBinding()] +param( + [string] $WorkDirectory, + [string] $SubscriptionId, + [string] $ResourceGroupName, + [string] $LogicAppName, + [string] $OrchestratorRunId, + [string] $AgentPath, + [string] $MetadataPath, + [string] $TenantId, + [string] $WorkflowDirectory, + [string] $ResumeReportPath, + [string[]] $KnownSecrets = @(), + [switch] $Zip +) + +function Protect-ScimText { + param([AllowEmptyString()][string] $Text, [string[]] $Secrets = @()) + $s = [System.Net.WebUtility]::HtmlDecode($Text) + # Logs can embed escaped JSON outside an otherwise valid JSON document. + for ($i = 0; $i -lt 3; $i++) { + $s = $s.Replace('\"', '"') + $s = [regex]::Replace($s, '\\u([0-9a-fA-F]{4})', { param($m) [string][char][Convert]::ToInt32($m.Groups[1].Value, 16) }) + } + foreach ($secret in $Secrets) { + if (-not [string]::IsNullOrEmpty($secret)) { + foreach ($literal in @($secret, [uri]::EscapeDataString($secret), [System.Net.WebUtility]::HtmlEncode($secret), [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($secret)))) { + $s = $s.Replace($literal, '[REDACTED]') + } + } + } + $s = [regex]::Replace($s, '(?i)\b(?:Bearer|Basic)\s+[A-Za-z0-9_.~+/=%-]+', '[REDACTED AUTH]') + $s = [regex]::Replace($s, '\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]*', '[REDACTED TOKEN]') + $s = [regex]::Replace($s, '(?i)(?:DefaultEndpointsProtocol|AccountKey|SharedAccessSignature|SharedAccessKey|Password|Pwd)\s*=[^\r\n]*', '[REDACTED CONNECTION/CREDENTIAL]') + $s = [regex]::Replace($s, '(?i)(?:Authorization|Proxy-Authorization|Set-Cookie|Cookie)\s*[:=]\s*[^\r\n]+', '[REDACTED HEADER]') + # OAuth code labels must not match diagnostic suffixes such as StatusCode. + $key = '(?:client[ _-]?secret|client[ _-]?assertion|assertion|password|passwd|secret|api[ _-]?key|subscription[ _-]?key|access[ _-]?token|id[ _-]?token|refresh[ _-]?token|token|sig|signature|(?"''])+', { + param($m) + $url = $m.Value + $url = [regex]::Replace($url, '^(https?://)[^/@\s]+:[^/@\s]+@', '$1[REDACTED]@') + $url = [regex]::Replace($url, '\?[^#]*', '?[REDACTED QUERY]') + [regex]::Replace($url, '#.*$', '#[REDACTED FRAGMENT]') + }) + return $s +} + +function Test-ScimSensitiveKey { + param([string] $Name) + $n = $Name -replace '[^a-zA-Z0-9]', '' + return $n -match '(?i)(authorization|cookie|apikey|subscriptionkey|clientsecret|password|passwd|accesstoken|idtoken|refreshtoken|assertion|connectionstring|accountkey|sharedaccess|credential|signature)' -or + $n -match '^(?i:secret|secrets|token|tokens|sig|key)$' +} + +function Protect-ScimValue { + param($Value, [string[]] $Secrets = @(), [string] $Name = '', [int] $Depth = 0) + if ($Depth -gt 80) { return '[OMITTED: depth limit]' } + if ((Test-ScimSensitiveKey $Name) -and $Name -ne 'rawCredentialsPersisted') { return '[REDACTED]' } + if ($null -eq $Value) { return $null } + if ($Value -is [string]) { + $s = Protect-ScimText $Value $Secrets + $trimmed = $Value.Trim() + if ($trimmed.StartsWith('{') -or $trimmed.StartsWith('[') -or $trimmed.StartsWith('"')) { + try { + $embedded = ConvertFrom-Json -InputObject $Value -AsHashtable -ErrorAction Stop + return (Protect-ScimValue $embedded $Secrets '' ($Depth + 1) | ConvertTo-Json -Depth 100 -Compress) + } catch { } + } + # Encoded form bodies and error strings often carry JSON or secrets one layer down. + if ($s -match '%[0-9a-fA-F]{2}') { + $decoded = [uri]::UnescapeDataString($s) + if ($decoded -ne $s) { return Protect-ScimValue $decoded $Secrets '' ($Depth + 1) } + } + return $s + } + if ($Value -is [System.Collections.IDictionary]) { + $result = [ordered]@{} + $tokenPayload = $Value.Contains('access_token') -or $Value.Contains('id_token') -or $Value.Contains('refresh_token') -or + $Value.Contains('client_secret') -or $Value.Contains('client_assertion') + if ($tokenPayload) { return '[OMITTED: credential or token payload]' } + foreach ($k in @($Value.Keys | Sort-Object -CaseSensitive)) { + $safeKey = Protect-ScimText ([string]$k) $Secrets + if ($Name -match '(?i)headers$' -and $k -notmatch '^(?i:content-type|content-length|date|retry-after|x-ms-request-id|x-ms-correlation-request-id|x-ms-client-tracking-id|x-ms-workflow-run-id|traceparent)$') { + $result[$safeKey] = '[REDACTED: non-allowlisted header]' + } else { + $result[$safeKey] = Protect-ScimValue $Value[$k] $Secrets ([string]$k) ($Depth + 1) + } + } + return $result + } + if ($Value -is [System.Collections.IEnumerable]) { + return ,@($Value | ForEach-Object { Protect-ScimValue $_ $Secrets '' ($Depth + 1) }) + } + return $Value +} + +function ConvertTo-ScimCanonicalJson { + param($Value) + function Order-Value($v) { + if ($v -is [System.Collections.IDictionary]) { + $o = [ordered]@{} + foreach ($k in @($v.Keys | Sort-Object -CaseSensitive)) { $o[$k] = Order-Value $v[$k] } + return $o + } + if ($v -is [System.Collections.IEnumerable] -and $v -isnot [string]) { + return ,@($v | ForEach-Object { Order-Value $_ }) + } + return $v + } + return (Order-Value $Value | ConvertTo-Json -Depth 100) +} + +function Get-ScimTiming { + param($Properties) + $result = @{ startUtc = $null; endUtc = $null; durationMilliseconds = $null; unavailableReason = $null } + try { + foreach ($field in @('startTime', 'endTime')) { + $value = $Properties.$field + if (-not $value) { continue } + # Newer PowerShell JSON parsers may return typed dates instead of strings. + $utc = if ($value -is [datetimeoffset]) { $value.UtcDateTime } + elseif ($value -is [datetime]) { + if ($value.Kind -eq [DateTimeKind]::Unspecified) { throw 'Timestamp has no explicit time zone.' } + $value.ToUniversalTime() + } else { + if ([string]$value -notmatch '(?i)(Z|[+-]\d{2}:\d{2})$') { throw 'Timestamp has no explicit time zone.' } + ([datetimeoffset]::Parse([string]$value, [Globalization.CultureInfo]::InvariantCulture)).UtcDateTime + } + if ($field -eq 'startTime') { $result.startUtc = $utc.ToString('o') } else { $result.endUtc = $utc.ToString('o') } + } + if ($result.startUtc -and $result.endUtc) { + $duration = ([datetimeoffset]::Parse($result.endUtc) - [datetimeoffset]::Parse($result.startUtc)).TotalMilliseconds + if ($duration -lt 0) { throw 'End precedes start.' } + $result.durationMilliseconds = $duration + } else { $result.unavailableReason = 'Start or end time was not supplied by the evidence source.' } + } catch { $result.unavailableReason = 'Invalid timing supplied by the evidence source.' } + return $result +} + +function Add-ScimEvidenceFailure { + param($Context, [string] $Source, [string] $Reason, [bool] $Required = $true) + $Context.failures.Add(@{ source = Protect-ScimText $Source $Context.secrets; reason = Protect-ScimText $Reason $Context.secrets; required = $Required }) +} + +function Get-ScimArmAuthentication { + param([string] $SubscriptionId, [string] $TenantId, [scriptblock] $InvokeCli) + $selector = if ($TenantId) { 'tenant' } else { 'subscription' } + if (-not $TenantId -and -not $SubscriptionId) { throw 'An explicit tenant or subscription is required for ARM authentication.' } + $commandArguments = @('account', 'get-access-token', '--resource', 'https://management.azure.com/', '--output', 'json', '--only-show-errors') + # Azure CLI rejects using --tenant and --subscription together. ARM resource + # URLs still select the requested subscription independently of token scope. + if ($TenantId) { $commandArguments += @('--tenant', $TenantId) } + else { $commandArguments += @('--subscription', $SubscriptionId) } + if (-not $InvokeCli) { + $InvokeCli = { + param([string[]] $CommandArguments) + $output = & az @CommandArguments 2>$null + return @{ exitCode = $LASTEXITCODE; output = $output } + } + } + $cliResult = & $InvokeCli $commandArguments + if ($cliResult.exitCode -ne 0) { throw "Azure CLI ARM access token acquisition failed (selector: $selector; CLI exit $($cliResult.exitCode))." } + try { $payload = ConvertFrom-Json -InputObject ($cliResult.output -join "`n") -AsHashtable -ErrorAction Stop } + catch { throw 'Azure CLI returned an invalid ARM authentication response; raw output is intentionally omitted.' } + if (-not $payload.accessToken) { throw 'Azure CLI returned no ARM access token.' } + if ($TenantId -and $payload.tenant -and $payload.tenant -ne $TenantId) { throw 'Azure CLI returned ARM authentication for a different tenant than requested.' } + $expiresUtc = $null + if ($payload.expires_on) { + try { $expiresUtc = [datetimeoffset]::FromUnixTimeSeconds([long]$payload.expires_on) } + catch { throw 'Azure CLI returned an invalid expires_on value for ARM authentication.' } + } + return @{ accessToken = $payload.accessToken; tenant = $payload.tenant; selector = $selector; expiresUtc = $expiresUtc } +} + +function New-ScimArmRequestState { + param($Authentication, [string] $SubscriptionId, [string] $TenantId, [scriptblock] $Acquire, [scriptblock] $Send, [scriptblock] $Clock) + if (-not $Acquire) { $Acquire = { param($subscription, $tenant) Get-ScimArmAuthentication -SubscriptionId $subscription -TenantId $tenant } } + if (-not $Clock) { $Clock = { [datetimeoffset]::UtcNow } } + if (-not $Send) { + $Send = { + param($uri, $headers, $receipt) + $response = Invoke-RestMethod -Uri $uri -Headers $headers -Method Get -MaximumRedirection 0 -TimeoutSec 60 -StatusCodeVariable responseStatus -ErrorAction Stop + if ($receipt) { $receipt.httpStatus = [int]$responseStatus } + return ,$response + } + } + $now = & $Clock + $secrets = [System.Collections.Generic.List[string]]::new() + if ($Authentication.accessToken) { $secrets.Add([string]$Authentication.accessToken) } + return @{ + authentication = $Authentication; subscriptionId = $SubscriptionId; tenantId = $TenantId + acquire = $Acquire; send = $Send; clock = $Clock; secretLiterals = $secrets + acquiredUtc = $now; nextProactiveRefreshUtc = $now; refreshCount = 0; retry401Count = 0 + } +} + +function Update-ScimArmAuthentication { + param($State) + $newAuthentication = & $State.acquire $State.subscriptionId $State.tenantId + if (-not $newAuthentication.accessToken) { throw 'ARM authentication refresh returned no access token.' } + $State.authentication = $newAuthentication + if (-not $State.secretLiterals.Contains([string]$newAuthentication.accessToken)) { $State.secretLiterals.Add([string]$newAuthentication.accessToken) } + $State.acquiredUtc = & $State.clock + # Azure CLI can return the same cached near-expiry token. Avoid repeatedly + # launching it for every request while retaining the bounded 401 recovery. + $State.nextProactiveRefreshUtc = $State.acquiredUtc.AddSeconds(30) + $State.refreshCount++ +} + +function Invoke-ScimRefreshingRequest { + param($State, [string] $Uri, [bool] $Arm, [System.Collections.IDictionary] $Receipt) + if (-not $Arm) { return ,(& $State.send $Uri @{} $Receipt) } + $now = & $State.clock + $expiry = $State.authentication.expiresUtc + $needsRefresh = -not $State.authentication.accessToken -or + ($expiry -and $expiry -le $now) -or + ($expiry -and $expiry -le $now.AddMinutes(5) -and $now -ge $State.nextProactiveRefreshUtc) -or + (-not $expiry -and $now -ge $State.acquiredUtc.AddMinutes(5)) + if ($needsRefresh) { Update-ScimArmAuthentication $State } + for ($attempt = 0; $attempt -lt 2; $attempt++) { + try { + $headers = @{ Authorization = ('Bearer ' + $State.authentication.accessToken) } + return ,(& $State.send $Uri $headers $Receipt) + } catch { + $status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { $_.Exception.Data['StatusCode'] } + if ($attempt -eq 0 -and $status -eq 401) { + $State.retry401Count++ + Update-ScimArmAuthentication $State + continue + } + throw + } + } +} + +function Invoke-ScimEvidenceRequest { + param($Context, [string] $Uri, [bool] $Arm = $true, [bool] $Required = $true, [System.Collections.IDictionary] $Receipt) + if ($null -eq $Receipt) { $Receipt = @{} } + $Receipt.succeeded = $false + $watch = if ($Context.requestMetricsEnabled) { [diagnostics.stopwatch]::StartNew() } else { $null } + $requestFailed = $false + try { + $u = [uri]$Uri + if ($u.Scheme -ne 'https' -or $u.UserInfo) { throw 'Evidence URI is not an approved HTTPS URI.' } + if ($Arm -and $u.Host -ne 'management.azure.com') { throw 'ARM pagination cannot leave management.azure.com.' } + if (-not $Arm -and $u.Host -notmatch '(^|\.)((azure\.com)|(windows\.net)|(azurewebsites\.net))$') { + throw 'Content link host is not an approved Azure evidence host.' + } + $response = & $Context.request $Uri $Arm $Receipt + if ($null -eq $response -and ($Arm -or $Receipt.httpStatus -lt 200 -or $Receipt.httpStatus -ge 300)) { + throw 'Evidence source returned an empty response without successful HTTP content-response proof.' + } + $Receipt.succeeded = $true + if ($response -is [System.Collections.IDictionary]) { return $response } + if ($null -eq $response) { return $null } + $normalized = ConvertTo-Json -InputObject $response -Depth 100 | ConvertFrom-Json -AsHashtable -NoEnumerate + return ,$normalized + } catch { + $requestFailed = $true + $Receipt.succeeded = $false + Add-ScimEvidenceFailure $Context $Uri $_.Exception.Message $Required + $failure = $Context.failures[$Context.failures.Count - 1] + $status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { $_.Exception.Data['StatusCode'] } + if ($status) { $failure.httpStatus = [int]$status } + if ($_.ErrorDetails.Message) { + $detail = $_.ErrorDetails.Message + try { $detail = ConvertFrom-Json -InputObject $detail -AsHashtable -ErrorAction Stop } catch { } + $failure.responseDetails = Protect-ScimValue $detail $Context.secrets + } + return $null + } finally { + if ($watch) { + $category = if (-not $Arm) { 'ContentPayload' } elseif ($Uri -match '/(?:scopeRepetitions|repetitions)/[^/?]+\?') { 'RepetitionDetail' } + elseif ($Uri -match '/(?:scopeRepetitions|repetitions)\?') { 'RepetitionList' } else { 'ArmMetadata' } + if (-not $Context.requestMetrics.Contains($category)) { + $Context.requestMetrics[$category] = @{ count = 0; failed = 0; totalMilliseconds = 0L; maximumMilliseconds = 0L } + } + $metric = $Context.requestMetrics[$category] + $metric.count++ + if ($requestFailed) { $metric.failed++ } + $metric.totalMilliseconds += $watch.ElapsedMilliseconds + $metric.maximumMilliseconds = [Math]::Max($metric.maximumMilliseconds, $watch.ElapsedMilliseconds) + } + } +} + +function Get-ScimEvidencePages { + param($Context, [string] $Uri, [bool] $Required = $true) + $items = [System.Collections.Generic.List[object]]::new() + $seen = [System.Collections.Generic.HashSet[string]]::new() + while ($Uri) { + if (-not $seen.Add($Uri)) { Add-ScimEvidenceFailure $Context $Uri 'Pagination cycle detected.' $Required; break } + $page = Invoke-ScimEvidenceRequest $Context $Uri $true $Required + if ($null -eq $page) { break } + if ($page -is [System.Collections.IDictionary] -and $page.Contains('value')) { + foreach ($entry in $page.value) { $items.Add($entry) } + $Uri = $page.nextLink + } elseif ($page -is [System.Collections.IEnumerable] -and $page -isnot [string] -and $page -isnot [System.Collections.IDictionary]) { + foreach ($entry in $page) { $items.Add($entry) } + $Uri = $null + } else { + Add-ScimEvidenceFailure $Context $Uri 'Collection response is neither a raw JSON array nor a value/nextLink envelope.' $Required + break + } + } + return ,@($items) +} + +function Get-ScimDefinitionActions { + param($Actions, [string] $Parent = '', [bool] $Repeated = $false) + foreach ($name in @($Actions.Keys | Sort-Object)) { + $a = $Actions[$name] + @{ name = $name; type = $a.type; parent = $Parent; repeated = $Repeated + targetWorkflow = $a.inputs.host.workflow.id; secureData = @($a.runtimeConfiguration.secureData.properties) } + $loop = $Repeated -or $a.type -in @('Foreach', 'Until') + if ($a.actions) { Get-ScimDefinitionActions $a.actions $name $loop } + if ($a.else.actions) { Get-ScimDefinitionActions $a.else.actions $name $loop } + if ($a.default.actions) { Get-ScimDefinitionActions $a.default.actions $name $loop } + foreach ($case in @($a.cases.Values)) { if ($case.actions) { Get-ScimDefinitionActions $case.actions $name $loop } } + } +} + +function Test-ScimRepetitionPayloadRequired { + param([string] $ActionType, [string] $ActionName, $Properties) + if ($Properties.status -ne 'Succeeded' -or $Properties.error) { return $true } + if ($ActionType -notin @('Http', 'Scope', 'Foreach', 'Until', 'Wait', 'If', 'Switch')) { return $true } + if ($ActionName -match '(?i)(Final_?TestResults|Evaluate_?Test_?Results|Test_?Outputs|Test_?Results?|Semantic|_Test$)') { return $true } + $output = $Properties.outputs + if ($output.headers.'x-ms-workflow-run-id') { return $true } + if ($output -is [System.Collections.IDictionary] -and $output.Contains('body')) { $output = $output.body } + if ($output -is [System.Collections.IDictionary]) { + foreach ($field in @('overallResult', 'testResult', 'result', 'testResults', 'authenticationModelResult')) { + if ($output.Contains($field)) { return $true } + } + } + return $false +} + +function Get-ScimActionEvidence { + param($Context, $Action, [string] $ActionUri, [bool] $Repeated = $false, [string[]] $SecureData = @(), [string] $ActionType = '', + [switch] $IsRepetition, [string] $RepetitionActionName) + $p = $Action.properties + $originalName = if ($RepetitionActionName) { $RepetitionActionName } else { $Action.name } + $cached = if ($Context.cachedActions) { $Context.cachedActions[$ActionUri] } else { $null } + $metadataOnly = $IsRepetition -and -not (Test-ScimRepetitionPayloadRequired $ActionType $originalName $p) + if ($metadataOnly -and $cached -and $cached.status -eq $p.status -and + ($cached.childRunId -or (Test-ScimRepetitionPayloadRequired $ActionType $originalName @{ status = $p.status; outputs = $cached.outputs }))) { + $metadataOnly = $false + } + $inputValue = $null; $outputValue = $null + if (-not $metadataOnly) { $inputValue = $p.inputs; $outputValue = $p.outputs } + $present = @{ + inputs = -not $metadataOnly -and $p -is [System.Collections.IDictionary] -and $p.Contains('inputs') + outputs = -not $metadataOnly -and $p -is [System.Collections.IDictionary] -and $p.Contains('outputs') + } + $presenceSources = @{ inputs = 'InlineProperty'; outputs = 'InlineProperty' } + $receipts = @{} + $reused = @{} + $canReuse = $cached -and $cached.contentAvailability -is [System.Collections.IDictionary] -and + $cached.status -eq $p.status -and $p.status -in @('Succeeded', 'Failed', 'Cancelled', 'TimedOut', 'Aborted') + foreach ($kind in @('inputs', 'outputs')) { + $link = $p["${kind}Link"] + if ($kind -in $SecureData -or $metadataOnly) { continue } + if ($link.uri -or ($canReuse -and -not $present[$kind] -and $cached.Contains($kind) -and $cached.contentAvailability[$kind].status -in @('Collected', 'ReusedSanitized'))) { + if ($canReuse -and $cached.contentAvailability[$kind].status -in @('Collected', 'ReusedSanitized') -and $cached.Contains($kind)) { + $value = $cached[$kind] + $reused[$kind] = $true + $present[$kind] = $true + $presenceSources[$kind] = 'CachedSanitizedPayload' + $Context.reusedPayloadCount++ + } else { + $receipt = @{} + $value = Invoke-ScimEvidenceRequest $Context $link.uri $false $true -Receipt $receipt + $receipts[$kind] = $receipt + $present[$kind] = $receipt.succeeded + $presenceSources[$kind] = 'ContentLinkResponse' + } + if ($kind -eq 'inputs') { $inputValue = $value } else { $outputValue = $value } + } + } + if ($outputValue -is [string] -and $outputValue -match '^\s*[\{\[]') { + try { $outputValue = ConvertFrom-Json -InputObject $outputValue -AsHashtable -NoEnumerate -ErrorAction Stop } catch { } + } + $serialized = @{ inputs = $inputValue; outputs = $outputValue } | ConvertTo-Json -Depth 100 -Compress + $sensitive = $originalName -match '(?i)(oauth|token|client.?assertion)' -or + $serialized -match '(?i)(/token\b|/oauth|access_token|refresh_token|id_token|client_secret|client_assertion|grant_type)' + $metadata = $Action + if ($metadataOnly) { + $metadata = @{} + foreach ($key in $Action.Keys) { $metadata[$key] = $Action[$key] } + $metadata.properties = @{} + foreach ($key in $p.Keys) { + if ($key -notin @('inputs', 'outputs')) { $metadata.properties[$key] = $p[$key] } + } + } + $entry = @{ + name = Protect-ScimValue $Action.name $Context.secrets; status = Protect-ScimValue $p.status $Context.secrets + code = Protect-ScimValue $p.code $Context.secrets; timing = Get-ScimTiming $p + semanticResult = 'Unknown'; semanticReason = 'Runtime status alone does not establish test success.' + evidence = Protect-ScimValue $metadata $Context.secrets + inputs = $null; outputs = $null; repetitions = @(); securityExclusions = @(); contentAvailability = @{} + payloadCollection = if ($metadataOnly) { 'MetadataOnlySuccessfulRepetition' } elseif ($IsRepetition) { 'RequiredDiagnosticRepetition' } else { 'TopLevelDiagnosticPayloads' } + } + # Discovery uses the raw header only in memory, before the allowlisted evidence is saved. + $childId = $outputValue.headers.'x-ms-workflow-run-id' + if (-not $childId -and $canReuse -and -not $metadataOnly) { $childId = $cached.childRunId } + if ($childId) { $entry.childRunId = Protect-ScimValue ([string]$childId) $Context.secrets } + if ($sensitive) { + $entry.inputs = '[OMITTED: sensitive authentication action payload]' + $entry.outputs = '[OMITTED: sensitive authentication action payload]' + $safeRetries = @($p.retryHistory | Where-Object { $null -ne $_ } | ForEach-Object { + Select-ScimMetadataFields $_ @('id', 'status', 'statusCode', 'code', 'startTime', 'endTime', 'trackingId', 'correlation') + }) + # Inline properties, retries and error details can also contain token endpoint payloads. + $entry.evidence = Protect-ScimValue @{ + name = $Action.name; properties = @{ + status = $p.status; code = $p.code; startTime = $p.startTime; endTime = $p.endTime + trackingId = $p.trackingId; clientTrackingId = $p.clientTrackingId + correlation = $p.correlation; inputsLink = $p.inputsLink; outputsLink = $p.outputsLink + repetitionIndexes = $p.repetitionIndexes; canResubmit = $p.canResubmit + retryHistory = $safeRetries + error = @{ code = $p.error.code; message = '[OMITTED: sensitive authentication action error detail]' } + } + } $Context.secrets + } else { + $entry.inputs = Protect-ScimValue $inputValue $Context.secrets + $entry.outputs = Protect-ScimValue $outputValue $Context.secrets + $semanticPayload = if ($outputValue -is [System.Collections.IDictionary] -and $outputValue.Contains('body')) { $outputValue.body } else { $outputValue } + if ($semanticPayload -is [System.Collections.IDictionary]) { + foreach ($field in @('overallResult', 'testResult', 'result')) { + if ($semanticPayload[$field] -is [string] -and $semanticPayload[$field] -match '^(?i:passed|failed|pass|fail|skipped|not_applicable|not_gallery_ready)$') { + $entry.semanticResult = $semanticPayload[$field] + $entry.semanticReason = "Explicit $field in the action output, not inferred from runtime status." + break + } + } + } + } + foreach ($kind in $SecureData) { + if ($kind -notin @('inputs', 'outputs')) { continue } + $entry[$kind] = '[EXCLUDED: workflow secureData protection]' + $entry.securityExclusions += @{ source = $kind; reason = 'Workflow definition intentionally protects this payload with runtimeConfiguration.secureData. Bytes are not claimed or fetched.' } + if ($entry.evidence.properties -is [System.Collections.IDictionary]) { + $entry.evidence.properties[$kind] = '[EXCLUDED: workflow secureData protection]' + $entry.evidence.properties.Remove("${kind}Link") + } + } + foreach ($kind in @('inputs', 'outputs')) { + $entry.contentAvailability[$kind] = if ($kind -in $SecureData) { + @{ status = 'SecurityExcluded'; reason = 'Intentionally protected by workflow secureData.' } + } elseif ($metadataOnly) { + @{ status = 'NotCollectedByScope'; reason = 'Successful repetition payload is intentionally outside FailureFocused diagnostic scope. All supplied metadata, timing and errors are retained; payload bytes are not claimed.' } + } elseif ($p.status -in @('Skipped', 'NotStarted') -and -not $present[$kind]) { + @{ status = 'NotExecuted'; reason = 'Skipped or not-started action has no execution payload; no payload bytes are claimed.' } + } elseif ($sensitive) { + @{ status = 'SecurityExcluded'; reason = 'Sensitive authentication action payload is deliberately omitted.' } + } elseif ($present[$kind]) { + @{ status = if ($reused[$kind]) { 'ReusedSanitized' } else { 'Collected' } + presenceSource = $presenceSources[$kind] + httpStatus = if ($receipts[$kind]) { $receipts[$kind].httpStatus } else { $null } + reason = if ($reused[$kind]) { 'Sanitized terminal payload reused from the previous report; current ARM metadata was read fresh.' } else { $null } } + } else { + @{ status = 'Unavailable'; reason = 'ARM did not provide inline content or a usable content link.' } + } + if ($IsRepetition -and $p.status -in @('Failed', 'TimedOut', 'Cancelled', 'Aborted') -and -not $sensitive -and + $ActionType -in @('Http', 'Workflow', 'Compose') -and $kind -notin $SecureData -and -not $present[$kind]) { + Add-ScimEvidenceFailure $Context "$ActionUri/$kind" 'Failed or interrupted repetition request/response payload is unavailable within FailureFocused diagnostic scope.' + } + } + if (-not $Repeated -and -not $metadataOnly -and -not $sensitive -and 'outputs' -notin $SecureData -and -not $present.outputs -and + $ActionType -in @('Http', 'Workflow', 'Compose') -and $p.status -eq 'Succeeded') { + Add-ScimEvidenceFailure $Context "$ActionUri/outputs" 'Successful data-producing action supplied no readable output evidence.' + } + if ($entry.timing.unavailableReason -and $p.status -notin @('Skipped', 'NotStarted')) { + Add-ScimEvidenceFailure $Context $ActionUri $entry.timing.unavailableReason + } + $entry.collectionState = 'InProgress' + $entry.type = Protect-ScimValue $ActionType $Context.secrets + if ($Context.activeWorkflow -and $ActionUri -eq $Context.activeActionUri) { + $Context.activeWorkflow.actions += $entry + } + if ($Repeated -and $ActionType -ne 'Until') { + $repetitionKind = if ($ActionType -eq 'Foreach') { 'scopeRepetitions' } else { 'repetitions' } + $reps = Get-ScimEvidencePages $Context "$ActionUri/${repetitionKind}?api-version=2022-03-01" + $entry.repetitionCollection = @{ kind = $repetitionKind } + foreach ($rep in @($reps | Sort-Object name)) { + $repUri = "$ActionUri/$repetitionKind/$([uri]::EscapeDataString($rep.name))" + $props = $rep.properties + $successfulMetadataOnly = $props -is [System.Collections.IDictionary] -and + -not (Test-ScimRepetitionPayloadRequired $ActionType $Action.name $props) + $listed = $props -is [System.Collections.IDictionary] -and ( + $props.status -in @('Skipped', 'NotStarted') -or + ($successfulMetadataOnly -and $props.startTime -and $props.endTime) -or ( + $props.status -in @('Succeeded', 'Failed', 'Cancelled', 'TimedOut', 'Aborted') -and + $props.startTime -and $props.endTime -and $props.Contains('code') -and + ($props.status -ne 'Failed' -or $props.Contains('error')) -and + ($props.Contains('inputs') -or $props.inputsLink.uri -or 'inputs' -in $SecureData) -and + ($props.Contains('outputs') -or $props.outputsLink.uri -or 'outputs' -in $SecureData) + )) + $detail = if ($listed) { $rep } else { Invoke-ScimEvidenceRequest $Context "${repUri}?api-version=2022-03-01" } + if ($detail) { + $repetition = Get-ScimActionEvidence $Context $detail $repUri $false $SecureData $ActionType -IsRepetition -RepetitionActionName $Action.name + $repetition.metadataSource = if ($listed) { 'RepetitionListing' } else { 'RepetitionDetail' } + $entry.repetitions += $repetition + $Context.completedRepetitionCount++ + if ($Context.checkpointPath) { Save-ScimCollectionCheckpoint $Context $Context.report $Context.checkpointPath -RepetitionUpdate } + } + } + } elseif ($ActionType -eq 'Until') { + $entry.iterationCount = Protect-ScimValue $p.iterationCount $Context.secrets + $entry.repetitionCollection = @{ kind = 'ContainedActionRepetitions'; reason = 'Runtime does not expose scope repetitions for an Until container. Its repeated child actions are collected through their repetitions endpoints.' } + } + if ($Context.currentWorkflow -eq 'Orchestrator_Workflow' -and $Action.name -eq 'Final_TestResults' -and -not $sensitive -and $null -ne $outputValue) { + $Context.finalResult = if ($outputValue.Contains('body')) { $outputValue.body } else { $outputValue } + } + $entry.collectionState = 'Completed' + return $entry +} + +function Get-ScimInvocationEvidence { + param($Entry, [string] $RepetitionPath = '') + if ($Entry.childRunId) { + @{ runId = $Entry.childRunId; repetitionPath = $RepetitionPath; evidence = $Entry } + } + foreach ($repetition in $Entry.repetitions) { + $path = if ($RepetitionPath) { "$RepetitionPath/$($repetition.name)" } else { [string]$repetition.name } + Get-ScimInvocationEvidence $repetition $path + } +} + +function Select-ScimMetadataFields { + param($Value, [string[]] $Names) + $selected = @{} + foreach ($name in $Names) { $selected[$name] = if ($Value -is [System.Collections.IDictionary]) { $Value[$name] } else { $null } } + return $selected +} + +function Add-ScimCachedAction { + param($Cache, $Entry, [string] $Uri) + $Cache[$Uri] = $Entry + $kind = if ($Entry.repetitionCollection.kind -in @('repetitions', 'scopeRepetitions')) { $Entry.repetitionCollection.kind } + elseif ($Entry.type -eq 'Foreach') { 'scopeRepetitions' } else { 'repetitions' } + foreach ($repetition in $Entry.repetitions) { + Add-ScimCachedAction $Cache $repetition "$Uri/$kind/$([uri]::EscapeDataString($repetition.name))" + } +} + +function Get-ScimSanitizedReportSnapshot { + param($Report, [string[]] $Secrets = @()) + # Workflow headers and each action payload are sanitized when collected. + # Reprocessing the growing inventory on every checkpoint is quadratic work. + $snapshot = @{} + foreach ($key in $Report.Keys) { + if ($key -ne 'workflows') { $snapshot[$key] = $Report[$key] } + } + $snapshot = Protect-ScimValue $snapshot $Secrets + $snapshot.workflows = $Report.workflows + return $snapshot +} + +function Save-ScimCollectionCheckpoint { + param($Context, $Report, [string] $Path, [switch] $RepetitionUpdate, [switch] $Force) + if (-not $Path) { return } + $completedActions = @($Report.workflows | ForEach-Object { $_.actions | Where-Object collectionState -ne 'InProgress' }).Count + if (-not $Force -and ($RepetitionUpdate -or $completedActions -ne 1) -and $Context.lastCheckpointUtc -and + [datetimeoffset]::UtcNow -lt $Context.lastCheckpointUtc.AddSeconds(30)) { return } + $Report.finalResults = $Context.finalResult + $Report.evidenceFailures = @($Context.failures) + $Report.runtime.evidenceRequests = $Context.requestMetrics + $Report.collection = @{ + status = 'InProgress'; completedActionCount = $completedActions + completedRepetitionCount = $Context.completedRepetitionCount + reason = 'Incremental evidence only; collection and readiness evaluation have not finished.' + } + $Report.completeness = @{ complete = $false; requiredUnavailableCount = @($Context.failures | Where-Object required).Count } + $Report.galleryReadiness = @{ ready = $false; status = 'Incomplete'; result = 'INCOMPLETE'; exitCode = 2; reason = $Report.collection.reason } + $safe = Get-ScimSanitizedReportSnapshot $Report $Context.secrets + $fullPath = [IO.Path]::GetFullPath($Path) + [void][IO.Directory]::CreateDirectory([IO.Path]::GetDirectoryName($fullPath)) + $pendingPath = "$fullPath.pending" + [IO.File]::WriteAllText($pendingPath, (ConvertTo-ScimCanonicalJson $safe) + "`n", [Text.UTF8Encoding]::new($false)) + [IO.File]::Move($pendingPath, $fullPath, $true) + $Context.lastCheckpointUtc = [datetimeoffset]::UtcNow + Write-Progress -Activity 'Collecting SCIM validation evidence' -Status "$($Report.collection.completedActionCount) actions, $($Context.completedRepetitionCount) repetitions saved; $($Report.workflows.Count) exact workflow runs" +} + +function Get-ScimRecordedTiming { + param($Properties) + $timing = Get-ScimTiming $Properties + $state = if (-not $Properties.startTime -and -not $Properties.endTime) { 'Unrecorded' } + elseif ($Properties.startTime -and -not $Properties.endTime) { 'Open' } + elseif (-not $Properties.startTime -and $Properties.endTime) { 'MissingStart' } + elseif ($timing.unavailableReason) { 'Invalid' } else { 'Complete' } + if ($timing.unavailableReason -like 'Invalid*') { $state = 'Invalid' } + return @{ timing = $timing; timingState = $state } +} + +function Get-ScimAgentEvents { + param($Context, [string] $Path, [string] $WorkDirectory) + $result = @{ + status = 'Unavailable'; sourceFileName = $null; unavailableReason = $null + events = @(); openEventCount = 0; missingTimingCount = 0; invalidEventCount = 0 + } + if (-not $Path) { + $result.unavailableReason = 'No agentEventsPath was supplied; historical agent/setup timings were not recorded.' + Add-ScimEvidenceFailure $Context 'Agent events' $result.unavailableReason $false + return $result + } + try { + if (-not [IO.Path]::IsPathRooted($Path)) { $Path = Join-Path $WorkDirectory $Path } + $result.sourceFileName = [IO.Path]::GetFileName($Path) + if ($result.sourceFileName -match '(?i)(parameters|appsettings)') { throw 'Configuration files are not accepted as agent event logs.' } + $raw = Get-Content -LiteralPath $Path -Raw -ErrorAction Stop + if (-not $raw.TrimStart().StartsWith('[')) { throw 'Agent event evidence must be a JSON array.' } + try { $events = ConvertFrom-Json -InputObject $raw -AsHashtable -NoEnumerate -ErrorAction Stop } + catch { throw 'Agent event JSON could not be parsed. Supply a valid array of event objects.' } + $index = 0 + foreach ($event in $events) { + $index++ + if ($event -isnot [System.Collections.IDictionary]) { + $result.events += @{ + index = $index; name = $null; kind = $null; parentName = $null; status = $null; details = $null + timing = Get-ScimTiming @{}; timingState = 'Invalid' + unavailableReason = 'Recorded event is not a JSON object.' + } + $result.invalidEventCount++ + continue + } + $record = Select-ScimMetadataFields $event @('name', 'kind', 'parentName', 'status', 'details') + $record.index = $index + $recorded = Get-ScimRecordedTiming $event + $record.timing = $recorded.timing + $record.timingState = $recorded.timingState + $record.unavailableReason = if (-not $event.name -or -not $event.kind -or -not $event.status) { + 'One or more explicit event name, kind or status fields were not recorded.' + } else { $record.timing.unavailableReason } + switch ($record.timingState) { + 'Open' { $result.openEventCount++ } + 'Invalid' { $result.invalidEventCount++ } + { $_ -in @('Unrecorded', 'MissingStart') } { $result.missingTimingCount++ } + } + $result.events += Protect-ScimValue $record $Context.secrets + } + if (-not $result.events.Count) { throw 'Agent event evidence contains no recorded events.' } + $partial = $result.openEventCount -gt 0 -or $result.missingTimingCount -gt 0 -or $result.invalidEventCount -gt 0 -or + @($result.events | Where-Object unavailableReason).Count -gt 0 + $result.status = if ($partial) { 'Partial' } else { 'Available' } + if ($partial) { + $result.unavailableReason = 'Some recorded agent events are open, invalid or missing fields/timings; no timestamps were invented.' + Add-ScimEvidenceFailure $Context 'Agent events' $result.unavailableReason $false + } + } catch { + $result.status = 'Unavailable' + $result.unavailableReason = Protect-ScimText $_.Exception.Message $Context.secrets + Add-ScimEvidenceFailure $Context 'Agent events' $result.unavailableReason $false + } + return $result +} + +function Get-ScimSemanticTests { + param($FinalResult, $Applicability, $Workflows, [string[]] $Methods = @()) + foreach ($test in @($FinalResult.testResults | Sort-Object testName)) { + if ($null -eq $test) { continue } + $app = $Applicability[$test.testName] + $testReason = if ($test.applicabilityReason) { $test.applicabilityReason } elseif ($test.reason) { $test.reason } else { $null } + $result = [string]$test.testResult + $method = if ($test.testName -eq 'Validate_Credentials_Test' -or $test.testName -match '(?i)oauth') { 'OAuth2ClientCredentials' } + elseif ($test.testName -match '(?i)(federat|workload.?identity|(^|_)WIF(_|$))') { 'WorkloadIdentityFederation' } + elseif ($test.testName -match '(?i)bearer.*connectivity|connectivity.*bearer') { 'Bearer' } else { $null } + if (-not $testReason -and $result -match '^(?i:NOT_APPLICABLE)\s*-\s*(.+)$') { $testReason = $Matches[1] } + $excluded = ($null -ne $app -and $app.applicable -eq $false -and -not [string]::IsNullOrWhiteSpace($app.reason)) -or + ($test.applicable -eq $false -and $result -match '^(?i:NOT_APPLICABLE)(?:\s*-|$)' -and -not [string]::IsNullOrWhiteSpace($testReason)) -or + ($method -and $Methods.Count -gt 0 -and $method -notin $Methods -and $result -match '^(?i:NOT_APPLICABLE)(?:\s*-|$)' -and $testReason) + $passed = $result -match '^(?i:pass|passed|success|succeeded|true)(?:\s*\([^)]*\))?$' + if ($method -and $Methods.Count -gt 0) { + $excluded = $method -notin $Methods + if ($excluded) { $testReason = 'Authentication method was not selected; recorded runtime result is retained as historical evidence.' } + } + $warningAllowed = $test.testCategory -match '(?i)(^|/)OptionalSuccess$' -and $result -match '^(?i:WARNING)\b' + $timings = @($Workflows | ForEach-Object { + $workflowName = $_.name + $_.actions | Where-Object { + (($_.name -replace '[^a-zA-Z0-9]', '') -eq ($test.testName -replace '[^a-zA-Z0-9]', '')) + } | ForEach-Object { @{ workflow = $workflowName; action = $_.name; timing = $_.timing; status = $_.status } } + }) + @{ + name = $test.testName; category = $test.testCategory; semanticResult = $result + applicable = -not $excluded; applicabilityReason = if ($excluded) { if ($method) { $testReason } elseif ($app.reason) { $app.reason } else { $testReason } } else { 'Applicable unless explicitly excluded with a reason.' } + passing = (-not $excluded) -and $passed; evidence = $test + authenticationMethod = $method; nonBlockingWarning = [bool]$warningAllowed + classification = if ($excluded) { 'NotApplicable' } elseif ($passed) { 'Passed' } elseif ($warningAllowed) { 'Warning' } elseif ($result -match '^(?i:FAILED|FAIL)\b') { 'Failed' } else { 'Unknown' } + scopeTimings = $timings + timingUnavailableReason = if (-not $timings.Count) { 'No exact test-name scope match; use the workflow action timing inventory.' } else { $null } + } + } +} + +function New-ScimValidationReport { + param( + [string] $SubscriptionId, [string] $ResourceGroupName, [string] $LogicAppName, + [string] $OrchestratorRunId, [string] $AgentPath, [string] $WorkflowDirectory, + [System.Collections.IDictionary] $Metadata = @{}, [string[]] $KnownSecrets = @(), + [scriptblock] $Request, [System.Collections.IDictionary] $RuntimeMetadata = @{}, + [string] $WorkDirectory = (Get-Location).Path, [System.Collections.IDictionary] $AuthenticationState, + [System.Collections.IDictionary] $PreviousReport, [string] $CheckpointPath + ) + foreach ($section in @('environment', 'harness', 'preflight')) { + if (-not $Metadata[$section]) { $Metadata[$section] = @{} } + } + $methods = if ($Metadata.supportedAuthenticationMethods) { @($Metadata.supportedAuthenticationMethods) } else { @($Metadata.authenticationMethods) } + $ctx = @{ + request = $Request; secrets = @($KnownSecrets) + @($Metadata.knownSecrets) + failures = [System.Collections.Generic.List[object]]::new(); finalResult = $null + securityExclusions = [System.Collections.Generic.List[object]]::new() + cachedActions = @{}; reusedPayloadCount = 0 + requestMetricsEnabled = [bool]$CheckpointPath -or [bool]$AuthenticationState; requestMetrics = @{} + completedRepetitionCount = 0; checkpointPath = $CheckpointPath; lastCheckpointUtc = $null + } + if ($AuthenticationState) { + foreach ($literal in @($KnownSecrets) + @($Metadata.knownSecrets)) { + if ($literal -and -not $AuthenticationState.secretLiterals.Contains($literal)) { $AuthenticationState.secretLiterals.Add($literal) } + } + $ctx.secrets = $AuthenticationState.secretLiterals + } + $report = @{ + schema = 'scim.validation-run-details'; schemaVersion = '1.1.0'; purpose = 'production-validation' + orchestratorRunId = if ($OrchestratorRunId) { $OrchestratorRunId } else { $null } + fullRun = @{ + status = if ($OrchestratorRunId) { 'Unavailable' } else { 'NotExecuted' } + timing = Get-ScimTiming $(if ($OrchestratorRunId) { @{} } else { $Metadata.preflight }) + unavailableReason = if ($OrchestratorRunId) { 'An execution ID was supplied, but Azure Orchestrator run evidence has not been obtained.' } else { $null } + } + provenance = @{ agent = $null; sourceVersion = @{ value = $null; sha256 = $null; fileName = 'VERSION'; scope = 'Local reporting source only; not the deployed historical run version'; unavailableReason = $null }; sourceGit = @{ revision = $null; dirty = $null; unavailableReason = $null }; workflows = @() } + environment = @{}; runtime = $RuntimeMetadata + harness = @{}; authentication = @{ selectedMethods = $methods; applicability = $Metadata.applicability; policy = 'Every selected supported authentication method must pass (AND), including static-token connectivity. At least one selected gallery-eligible method is required.' } + workflows = @(); tests = @(); finalResults = $null; provisioningLogs = @(); childRunAssociations = @() + suppliedSource = @{ attribution = 'Caller-supplied; not independently verified'; metadata = $null } + deploymentOverlay = @{ attribution = 'Caller-supplied identifiers/hashes; no deployment equivalence is asserted'; metadata = $null } + evidenceFailures = @(); completeness = @{}; galleryReadiness = @{} + redaction = @{ policyVersion = '1'; rawCredentialsPersisted = $false; tokenPayloads = 'Omitted'; headers = 'Allowlisted'; urlQueries = 'All values omitted' } + diagnosticScope = @{ + name = 'FailureFocused' + metadata = 'Every action and repetition, including all supplied timing, status, errors and retry metadata; no sampling or truncation.' + payloads = 'Top-level actions and failed, timed-out, cancelled or aborted repetitions; successful semantic-result producers and child-workflow calls remain included.' + successfulRepeatedPollPayloads = 'Intentionally not collected. NotCollectedByScope is not a payload-download failure and does not claim payload completeness.' + } + } + $report.historicalRun = @{ attribution = 'Caller-recorded deployed-run claims; not inferred from current source'; metadata = $null; unavailableReason = 'Historical deployed version and authentication policy were not recorded.' } + if ($Metadata.historicalRun) { + $report.historicalRun.metadata = Select-ScimMetadataFields $Metadata.historicalRun @('version', 'authenticationPolicy', 'selectedMethods', 'description') + if ($Metadata.historicalRun.sourceVersion) { + $report.historicalRun.metadata.sourceVersion = Select-ScimMetadataFields $Metadata.historicalRun.sourceVersion @('value', 'sha256') + } + $report.historicalRun.unavailableReason = $null + } elseif ($PreviousReport.historicalRun.metadata) { + $report.historicalRun = $PreviousReport.historicalRun + } + try { + $versionPath = Join-Path $WorkflowDirectory 'VERSION' + $sourceVersion = (Get-Content -LiteralPath $versionPath -Raw -ErrorAction Stop).Trim() + if ($sourceVersion -notmatch '^\d+\.\d+(?:\.\d+)?$') { throw 'VERSION must contain a numeric release version.' } + $report.provenance.sourceVersion.value = $sourceVersion + $report.provenance.sourceVersion.sha256 = (Get-FileHash -LiteralPath $versionPath -Algorithm SHA256 -ErrorAction Stop).Hash.ToLowerInvariant() + } catch { + $report.provenance.sourceVersion.unavailableReason = 'Local VERSION is missing, unreadable, or invalid.' + Add-ScimEvidenceFailure $ctx 'VERSION provenance' $report.provenance.sourceVersion.unavailableReason + } + $report.provenance.comparison = @{ + deployedVersionMatchesSource = if ($report.historicalRun.metadata.version -and $report.provenance.sourceVersion.value) { $report.historicalRun.metadata.version -eq $report.provenance.sourceVersion.value } else { $null } + reason = 'Local VERSION, agent and workflow hashes describe current source, not the definitions or authentication policy deployed for this run.' + } + $agentWindow = if ($Metadata.agentRun) { $Metadata.agentRun } else { @{} } + $recordedAgentWindow = Get-ScimRecordedTiming $agentWindow + $report.agentRun = @{ + status = $agentWindow.status; timing = $recordedAgentWindow.timing; timingState = $recordedAgentWindow.timingState + statusUnavailableReason = if (-not $agentWindow.status) { 'Agent run status was not explicitly recorded.' } else { $null } + window = 'Explicit agent/setup-to-cleanup window; separate from Azure Orchestrator fullRun timing.' + } + $report.agentEvents = Get-ScimAgentEvents $ctx $Metadata.agentEventsPath $WorkDirectory + if ($Metadata.source) { $report.suppliedSource.metadata = Select-ScimMetadataFields $Metadata.source @('repositoryUrl', 'revision', 'branch', 'dirty', 'description') } + if ($Metadata.deploymentOverlay) { + $report.deploymentOverlay.metadata = Select-ScimMetadataFields $Metadata.deploymentOverlay @('name', 'revision', 'sha256', 'description') + $report.deploymentOverlay.metadata.workflowHashes = @($Metadata.deploymentOverlay.workflowHashes | ForEach-Object { + Select-ScimMetadataFields $_ @('name', 'sha256') + }) + } + foreach ($field in @('tenantId', 'subscriptionId', 'resourceGroupName', 'logicAppName', 'managedIdentity', 'entraApplication', 'servicePrincipal', 'provisioningJob', 'endpoint', 'operatorIdentity')) { + $report.environment[$field] = @{ value = $Metadata.environment[$field]; unavailableReason = if ($null -eq $Metadata.environment[$field]) { 'Not supplied or obtainable with the selected evidence permissions.' } else { $null } } + } + foreach ($pair in @(@('subscriptionId', $SubscriptionId), @('resourceGroupName', $ResourceGroupName), @('logicAppName', $LogicAppName))) { + if ($pair[1]) { $report.environment[$pair[0]] = @{ value = $pair[1]; unavailableReason = $null } } + } + foreach ($field in @('name', 'version', 'model', 'reasoning', 'context')) { + $report.harness[$field] = @{ value = $Metadata.harness[$field]; unavailableReason = if ($null -eq $Metadata.harness[$field]) { 'Not explicitly supplied by caller; hidden host data is not inspected.' } else { $null } } + } + try { + $agent = Get-Item -LiteralPath $AgentPath -ErrorAction Stop + $report.provenance.agent = @{ fileName = $agent.Name; sha256 = (Get-FileHash -LiteralPath $agent.FullName -Algorithm SHA256).Hash.ToLowerInvariant() } + $report.provenance.agent.sourceVersion = $report.provenance.sourceVersion.value + $report.provenance.agent.versionSource = 'VERSION; local source only' + $rev = & git -C $agent.DirectoryName rev-parse HEAD 2>$null + if ($LASTEXITCODE -ne 0) { throw 'Agent directory is not an accessible Git worktree.' } + $dirty = @(& git -C $agent.DirectoryName status --porcelain --untracked-files=normal 2>$null) + if ($LASTEXITCODE -ne 0) { throw 'Git dirty state could not be read.' } + $report.provenance.sourceGit.revision = [string]$rev + $report.provenance.sourceGit.dirty = $dirty.Count -gt 0 + } catch { + $report.provenance.sourceGit.unavailableReason = $_.Exception.Message + Add-ScimEvidenceFailure $ctx 'Agent/source provenance' $_.Exception.Message + } + $definitions = @{} + $expectedTests = @() + foreach ($file in @(Get-ChildItem -LiteralPath $WorkflowDirectory -Filter '*_Workflow.json' -File -ErrorAction SilentlyContinue | Sort-Object Name)) { + try { + $wf = Get-Content -LiteralPath $file.FullName -Raw -ErrorAction Stop | ConvertFrom-Json -AsHashtable -ErrorAction Stop + $definitions[$file.BaseName] = @(Get-ScimDefinitionActions $wf.definition.actions) + if ($file.BaseName -eq 'Orchestrator_Workflow') { + $expectedTests = @($wf.definition.actions.Final_TestResults.inputs.testResults | ForEach-Object testName | Where-Object { $_ -and $_ -notmatch '^@' }) + } + $report.provenance.workflows += @{ name = $file.BaseName; sha256 = (Get-FileHash -LiteralPath $file.FullName -Algorithm SHA256).Hash.ToLowerInvariant(); source = 'Local definition; not a claim of deployed content equivalence' } + } catch { Add-ScimEvidenceFailure $ctx "Workflow definition $($file.Name)" $_.Exception.Message } + } + if (-not $definitions.Count) { Add-ScimEvidenceFailure $ctx 'Workflow definitions' 'No local workflow definitions were found.' } + $base = "https://management.azure.com/subscriptions/$([uri]::EscapeDataString($SubscriptionId))/resourceGroups/$([uri]::EscapeDataString($ResourceGroupName))/providers/Microsoft.Web/sites/$([uri]::EscapeDataString($LogicAppName))" + $api = "$base/hostruntime/runtime/webhooks/workflow/api/management" + if ($PreviousReport) { + if ($PreviousReport.schema -ne 'scim.validation-run-details' -or $PreviousReport.orchestratorRunId -ne $OrchestratorRunId -or + $PreviousReport.environment.subscriptionId.value -ne $SubscriptionId -or + $PreviousReport.environment.resourceGroupName.value -ne $ResourceGroupName -or + $PreviousReport.environment.logicAppName.value -ne $LogicAppName) { + throw 'Resume report does not match the requested schema, root run and Azure site.' + } + foreach ($previousWorkflow in $PreviousReport.workflows) { + if ($previousWorkflow.status -notin @('Succeeded', 'Failed', 'Cancelled', 'TimedOut', 'Aborted')) { continue } + $previousUri = "$api/workflows/$([uri]::EscapeDataString($previousWorkflow.name))/runs/$([uri]::EscapeDataString($previousWorkflow.runId))" + foreach ($previousAction in $previousWorkflow.actions) { + Add-ScimCachedAction $ctx.cachedActions $previousAction "$previousUri/actions/$([uri]::EscapeDataString($previousAction.name))" + } + } + } + $ctx.report = $report + Save-ScimCollectionCheckpoint $ctx $report $CheckpointPath -Force + if ($OrchestratorRunId) { + $site = Invoke-ScimEvidenceRequest $ctx "${base}?api-version=2022-03-01" + if ($site) { $report.environment.managedIdentity = @{ value = $site.identity; unavailableReason = if (-not $site.identity) { 'Site has no managed identity.' } else { $null } } } + $queue = [System.Collections.Generic.Queue[object]]::new() + $queue.Enqueue(@{ name = 'Orchestrator_Workflow'; runId = $OrchestratorRunId; discovery = 'Explicit OrchestratorRunId' }) + foreach ($supplied in @($Metadata.workflowRuns)) { + if ($null -eq $supplied) { continue } + if (-not $supplied.workflowName -or -not $supplied.runId) { + Add-ScimEvidenceFailure $ctx 'Supplied workflowRuns' 'Each supplied workflow run requires workflowName and runId.' + continue + } + $queue.Enqueue(@{ name = [string]$supplied.workflowName; runId = [string]$supplied.runId; discovery = 'Caller-supplied exact run ID' }) + $association = Select-ScimMetadataFields $supplied @('workflowName', 'runId', 'parentWorkflowName', 'parentRunId', 'parentActionName', 'repetitionName') + $association.attribution = 'Caller-supplied exact association; not inferred from hidden payloads' + $report.childRunAssociations += $association + } + $seenRuns = [System.Collections.Generic.HashSet[string]]::new() + $seenCorrelations = [System.Collections.Generic.HashSet[string]]::new() + while ($queue.Count) { + $item = $queue.Dequeue() + if (-not $seenRuns.Add("$($item.name)/$($item.runId)")) { continue } + $runUri = "$api/workflows/$([uri]::EscapeDataString($item.name))/runs/$([uri]::EscapeDataString($item.runId))" + $run = Invoke-ScimEvidenceRequest $ctx "${runUri}?api-version=2022-03-01" + if (-not $run) { continue } + $ctx.currentWorkflow = $item.name + $wfReport = @{ + name = $item.name; runId = $item.runId; status = $run.properties.status + discovery = $item.discovery + timing = Get-ScimTiming $run.properties; run = Protect-ScimValue $run $ctx.secrets + triggers = @(); actions = @(); scopes = @() + } + $wfReport = Protect-ScimValue $wfReport $ctx.secrets + $report.workflows += $wfReport + $ctx.activeWorkflow = $wfReport + if ($wfReport.timing.unavailableReason) { Add-ScimEvidenceFailure $ctx $runUri $wfReport.timing.unavailableReason } + if ($item.name -eq 'Orchestrator_Workflow') { $report.fullRun = @{ status = $wfReport.status; timing = $wfReport.timing } } + # The run header identifies the exact trigger history; never query an unrelated latest history. + if ($run.properties.trigger.name) { + $trigger = $run.properties.trigger + $wfReport.triggers = @(Get-ScimActionEvidence $ctx @{ name = $trigger.name; properties = $trigger } "$runUri/trigger") + if ($trigger.originHistoryName) { + $triggerUri = "$api/workflows/$([uri]::EscapeDataString($item.name))/triggers/$([uri]::EscapeDataString($trigger.name))/histories/$([uri]::EscapeDataString($trigger.originHistoryName))" + $history = Invoke-ScimEvidenceRequest $ctx "${triggerUri}?api-version=2022-03-01" + if ($history) { $wfReport.triggers += Get-ScimActionEvidence $ctx $history $triggerUri } + } + } else { Add-ScimEvidenceFailure $ctx "$runUri/trigger" 'Run header did not supply trigger evidence.' } + $actions = Get-ScimEvidencePages $ctx "$runUri/actions?api-version=2022-03-01" + if (-not $actions.Count) { Add-ScimEvidenceFailure $ctx "$runUri/actions" 'No action evidence was returned.' } + foreach ($action in @($actions | Sort-Object name)) { + $definition = @($definitions[$item.name] | Where-Object name -eq $action.name) | Select-Object -First 1 + $repeated = $definition.repeated -or $definition.type -eq 'Foreach' -or [bool]$action.properties.repetitionIndexes + $actionUri = "$runUri/actions/$([uri]::EscapeDataString($action.name))" + $ctx.activeActionUri = $actionUri + $entry = Get-ScimActionEvidence $ctx $action $actionUri $repeated $definition.secureData $definition.type + $entry.parentScope = Protect-ScimValue $definition.parent $ctx.secrets + $ctx.activeActionUri = $null + Save-ScimCollectionCheckpoint $ctx $report $CheckpointPath + if ($definition.type -eq 'Scope') { $wfReport.scopes += @{ name = $entry.name; status = $entry.status; timing = $entry.timing; semanticResult = $entry.semanticResult; semanticReason = $entry.semanticReason } } + if ($definition.type -eq 'Workflow') { + $childName = if ($definition.targetWorkflow) { ([string]$definition.targetWorkflow -split '/')[-1] } else { $action.name -replace '^Call_', '' } + $invocations = @(Get-ScimInvocationEvidence $entry) + foreach ($invocation in $invocations) { + $queue.Enqueue(@{ name = $childName; runId = $invocation.runId; discovery = 'Exact Workflow action output run-ID header' }) + $report.childRunAssociations += @{ + workflowName = $childName; runId = $invocation.runId; parentWorkflowName = $item.name + parentRunId = $item.runId; parentActionName = $action.name; repetitionName = $invocation.repetitionPath + attribution = 'Exact Workflow action output run-ID header' + } + } + if (-not $invocations.Count -and 'outputs' -in $definition.secureData -and $action.properties.status -ne 'Skipped') { + $ctx.securityExclusions.Add(@{ + source = $actionUri; reason = 'Workflow call outputs, including a possible child run-ID header, are intentionally secured. No exact child mapping is invented.' + }) + $correlation = if ($action.properties.correlation.clientTrackingId) { $action.properties.correlation.clientTrackingId } else { $run.properties.correlation.clientTrackingId } + if ($correlation -and $seenCorrelations.Add("$childName/$correlation")) { + $filter = [uri]::EscapeDataString("ClientTrackingId eq '$(([string]$correlation).Replace("'", "''"))'") + $childRunsUri = "$api/workflows/$([uri]::EscapeDataString($childName))/runs?api-version=2022-03-01&`$filter=$filter" + $correlatedRuns = Get-ScimEvidencePages $ctx $childRunsUri $false + foreach ($candidate in $correlatedRuns) { + if ($candidate.name -and $candidate.properties.correlation.clientTrackingId -eq $correlation) { + $queue.Enqueue(@{ name = $childName; runId = [string]$candidate.name; discovery = 'Correlation-only inventory; not an exact parent-action association' }) + } + } + } + } elseif (-not $invocations.Count -and $action.properties.status -notin @('Skipped', 'NotStarted')) { + Add-ScimEvidenceFailure $ctx $actionUri 'Child workflow run ID was not present in the call outputs.' + } + } + } + Save-ScimCollectionCheckpoint $ctx $report $CheckpointPath -Force + } + foreach ($expected in @('Orchestrator_Workflow', 'Initialization_Workflow', 'UserTests_Workflow', 'GroupTests_Workflow', 'SCIMTests_Workflow')) { + if (-not @($report.workflows | Where-Object name -eq $expected).Count) { Add-ScimEvidenceFailure $ctx $expected 'Required workflow run was not discovered or could not be read.' } + } + } else { + $report.fullRun.status = 'Blocked' + $report.fullRun.reason = $Metadata.preflight.reason + Add-ScimEvidenceFailure $ctx 'Execution' 'No Orchestrator run ID supplied; no execution is claimed.' + } + $report.finalResults = $ctx.finalResult + if (-not $methods.Count) { + $methods = if ($ctx.finalResult.supportedAuthenticationMethods) { @($ctx.finalResult.supportedAuthenticationMethods) } else { @($ctx.finalResult.authenticationModelResult.supportedAuthenticationMethods) } + $report.authentication.selectedMethods = $methods + } + $transport = if ($Metadata.scimAuthenticationMethod) { $Metadata.scimAuthenticationMethod } else { $ctx.finalResult.scimAuthenticationMethod } + $report.authentication.scimRequestTransport = @{ + value = $transport + unavailableReason = if (-not $transport) { 'Direct SCIM request transport was not supplied by caller or Final_TestResults.' } else { $null } + } + $report.authentication.selectionSource = if ($Metadata.supportedAuthenticationMethods -or $Metadata.authenticationMethods) { 'Caller-supplied selection for this assessment' } else { 'Recorded Final_TestResults' } + $report.authentication.recordedSelectedMethods = if ($ctx.finalResult.supportedAuthenticationMethods) { @($ctx.finalResult.supportedAuthenticationMethods) } else { @($ctx.finalResult.authenticationModelResult.supportedAuthenticationMethods) } + $report.authentication.selectionMismatch = $report.authentication.recordedSelectedMethods.Count -gt 0 -and + (($methods | Sort-Object -Unique) -join ',') -ne (($report.authentication.recordedSelectedMethods | Sort-Object -Unique) -join ',') + if ($report.authentication.selectionMismatch) { + Add-ScimEvidenceFailure $ctx 'Authentication selection provenance' 'Caller selection differs from recorded executed selection; reconcile the evidence rather than waive a selected method.' + } + if (-not $methods.Count) { Add-ScimEvidenceFailure $ctx 'Authentication selection' 'Selected authentication methods were not supplied by caller or Final_TestResults.' } + if (-not $ctx.finalResult.testResults) { Add-ScimEvidenceFailure $ctx 'Final_TestResults' 'Semantic test results were not available.' } + $applicability = if ($Metadata.applicability) { $Metadata.applicability } else { @{} } + $report.tests = @(Get-ScimSemanticTests $ctx.finalResult $applicability $report.workflows $methods) + $report.testSummary = @{ + total = $report.tests.Count + passed = @($report.tests | Where-Object classification -eq 'Passed').Count + warnings = @($report.tests | Where-Object classification -eq 'Warning').Count + failed = @($report.tests | Where-Object classification -eq 'Failed').Count + notApplicable = @($report.tests | Where-Object classification -eq 'NotApplicable').Count + unknown = @($report.tests | Where-Object classification -eq 'Unknown').Count + } + foreach ($expected in $expectedTests) { + if (@($report.tests | Where-Object name -eq $expected).Count -ne 1) { + Add-ScimEvidenceFailure $ctx "Semantic test $expected" 'Expected exactly one result from the local Orchestrator definition; result is missing or duplicated.' + } + } + foreach ($workflow in $report.workflows) { + foreach ($scope in $workflow.scopes) { + $semantic = @($report.tests | Where-Object { ($_.name -replace '[^a-zA-Z0-9]', '') -eq ($scope.name -replace '[^a-zA-Z0-9]', '') }) + if ($semantic.Count -eq 1) { $scope.semanticResult = Protect-ScimValue $semantic[0].semanticResult $ctx.secrets; $scope.semanticReason = 'Matched exact normalized test name in Final_TestResults.' } + } + } + if ($Metadata.provisioningLogPath) { + try { + if ([IO.Path]::GetFileName($Metadata.provisioningLogPath) -match '(?i)(parameters|appsettings).*\.json$') { throw 'Configuration files are not accepted as provisioning logs.' } + $log = Get-Content -LiteralPath $Metadata.provisioningLogPath -Raw -ErrorAction Stop + $report.provisioningLogs = @(@{ fileName = [IO.Path]::GetFileName($Metadata.provisioningLogPath); content = Protect-ScimValue $log $ctx.secrets }) + } catch { Add-ScimEvidenceFailure $ctx 'Supplied provisioning log' $_.Exception.Message } + } else { + Add-ScimEvidenceFailure $ctx 'Provisioning logs' 'No provisioning log was supplied; this optional source was not collected.' $false + } + $report.evidenceFailures = @($ctx.failures) + $missing = @($ctx.failures | Where-Object required).Count + $report.securityExclusions = @($ctx.securityExclusions) + $securedPayloadCount = @($report.workflows | ForEach-Object { $_.actions | ForEach-Object { $_.securityExclusions } }).Count + $report.completeness = @{ + complete = $missing -eq 0; requiredUnavailableCount = $missing + optionalUnavailableCount = @($ctx.failures | Where-Object { -not $_.required }).Count + securityExcludedCount = $ctx.securityExclusions.Count + $securedPayloadCount + scope = 'Complete within FailureFocused diagnostic scope means all required metadata/timings and in-scope payload evidence was collected. Successful repeated-poll payloads marked NotCollectedByScope are intentionally not collected. Security exclusions are separate; completeness never claims every payload or hidden parent-action association.' + } + $applicableTests = @($report.tests | Where-Object applicable) + $authTests = @($report.tests | Where-Object authenticationMethod) + $report.authentication.bearerConnectivity = $ctx.finalResult.BearerConnectivityTestOutputs + if ($ctx.finalResult.BearerConnectivityTestOutputs -and -not @($authTests | Where-Object authenticationMethod -eq 'Bearer').Count) { + $bearerResult = $ctx.finalResult.BearerConnectivityTestOutputs + $bearerSemantic = if ($bearerResult.overallResult) { $bearerResult.overallResult } else { $bearerResult.result } + $authTests += @(Get-ScimSemanticTests @{ testResults = @(@{ + testName = 'Bearer_Connectivity_Test'; testCategory = 'Authentication'; testResult = $bearerSemantic + details = $bearerResult + }) } @{} $report.workflows $methods) + } + $failedTests = @($applicableTests | Where-Object { -not $_.passing -and -not $_.nonBlockingWarning -and $_ -notin $authTests }) + $eligibleMethods = @('OAuth2ClientCredentials', 'WorkloadIdentityFederation') + $methodResults = @( + foreach ($method in @($methods | Sort-Object -Unique)) { + $results = @($authTests | Where-Object authenticationMethod -eq $method) + $supported = $method -in (@('Bearer') + $eligibleMethods) + $passing = $supported -and $results.Count -gt 0 -and @($results | Where-Object { -not $_.applicable -or -not $_.passing }).Count -eq 0 + if ($supported -and (-not $results.Count -or @($results | Where-Object { $_.classification -in @('Unknown', 'NotApplicable') -or $_.semanticResult -match '(?i)(NOT_APPLICABLE|MISSING|NOT_CONFIGURED|BLOCKED|CREDENTIAL)' }).Count)) { + Add-ScimEvidenceFailure $ctx "Selected authentication method $method" 'Selected method lacks a conclusive required result or required credentials; it cannot be waived as not applicable.' + } + @{ method = $method; eligible = $method -in $eligibleMethods; supported = $supported; passing = $passing; resultCount = $results.Count + reason = if ($passing) { 'All selected-method tests passed.' } elseif (-not $supported) { 'This selected method is unsupported.' } else { 'Every selected method, including static-token connectivity, must pass; aggregate success cannot override this result.' } } + } + ) + $galleryAuthPass = @($methodResults | Where-Object eligible).Count -gt 0 -and @($methodResults | Where-Object { -not $_.passing }).Count -eq 0 + $report.authentication.methodResults = $methodResults + $report.authentication.semanticResult = $ctx.finalResult.authenticationModelResult + if ($null -eq $ctx.finalResult.authenticationModelResult) { + Add-ScimEvidenceFailure $ctx 'Authentication semantics' 'Final_TestResults did not supply authenticationModelResult.' + } + $missing = @($ctx.failures | Where-Object required).Count + $report.evidenceFailures = @($ctx.failures) + $report.completeness.complete = $missing -eq 0 + $report.completeness.requiredUnavailableCount = $missing + $runtimeFailures = @($report.workflows | Where-Object { $_.name -eq 'Orchestrator_Workflow' -and $_.status -ne 'Succeeded' }) + $overallPassed = [string]$ctx.finalResult.overallResult -match '^(?i:pass|passed|success|succeeded|true)$' + $authOverall = if ($ctx.finalResult.authenticationModelResult -is [System.Collections.IDictionary]) { + if ($ctx.finalResult.authenticationModelResult.overallResult) { $ctx.finalResult.authenticationModelResult.overallResult } else { $ctx.finalResult.authenticationModelResult.result } + } else { $ctx.finalResult.authenticationModelResult } + $authPassed = [string]$authOverall -match '^(?i:pass|passed|success|succeeded|true)$' + $report.authentication.recordedAggregateResult = $authOverall + $report.authentication.recordedPolicy = if ($ctx.finalResult.authenticationPolicy) { $ctx.finalResult.authenticationPolicy } else { $ctx.finalResult.authenticationModelResult.authenticationPolicy } + $report.authentication.aggregatePolicyMismatch = $authPassed -and -not $galleryAuthPass + $report.provenance.executedVersion = @{ + value = if ($ctx.finalResult.LogicAppVersion) { $ctx.finalResult.LogicAppVersion } elseif ($ctx.finalResult.logicAppVersion) { $ctx.finalResult.logicAppVersion } else { $report.historicalRun.metadata.version } + attribution = if ($ctx.finalResult.LogicAppVersion) { 'Recorded Final_TestResults output' } else { 'Caller historicalRun metadata, when supplied; never inferred from local VERSION' } + } + if ($report.provenance.executedVersion.value) { + $report.provenance.comparison.deployedVersionMatchesSource = $report.provenance.executedVersion.value -eq $report.provenance.sourceVersion.value + } + $coreTests = @($applicableTests | Where-Object { $_ -notin $authTests }) + $ready = $missing -eq 0 -and $overallPassed -and $galleryAuthPass -and $authPassed -and $coreTests.Count -gt 0 -and $failedTests.Count -eq 0 -and $runtimeFailures.Count -eq 0 + $report.galleryReadiness = @{ + ready = $ready; status = if ($missing) { 'Incomplete' } elseif ($ready) { 'Ready' } else { 'NotReady' } + result = if ($missing) { 'INCOMPLETE' } elseif ($ready) { 'GALLERY_READY' } else { 'NOT_GALLERY_READY' } + reason = if ($missing) { 'Required evidence is unavailable; do not claim gallery readiness.' } elseif ($ready) { 'Core tests and every selected authentication method passed, including at least one gallery-eligible method.' } else { 'Core validation or a selected authentication method has not passed. Static-token connectivity alone is not gallery-ready.' } + failedOrUnknownApplicableTests = @($failedTests.name) + nonBlockingWarningTests = @($report.tests | Where-Object nonBlockingWarning | ForEach-Object name) + exitCode = if ($missing) { 2 } elseif ($ready) { 0 } else { 1 } + } + if ($AuthenticationState) { + $report.runtime.armTokenRefreshCount = $AuthenticationState.refreshCount + $report.runtime.arm401RetryCount = $AuthenticationState.retry401Count + } + $report.resume = @{ + requested = [bool]$PreviousReport; reusedPayloadCount = $ctx.reusedPayloadCount + policy = 'Only available sanitized terminal-action payloads are reused. ARM run/action/repetition metadata is read fresh and all failures are recomputed.' + } + $report.collection = @{ status = 'Completed'; completedActionCount = @($report.workflows | ForEach-Object { $_.actions }).Count } + if ($ctx.requestMetricsEnabled) { $report.runtime.evidenceRequests = $ctx.requestMetrics } + if ($CheckpointPath) { Write-Progress -Activity 'Collecting SCIM validation evidence' -Completed } + return Get-ScimSanitizedReportSnapshot $report $ctx.secrets +} + +function Complete-ScimValidationReport { + param([Parameter(Mandatory)][string] $ReportPath) + $report = Get-Content -LiteralPath $ReportPath -Raw -ErrorAction Stop | ConvertFrom-Json -AsHashtable -ErrorAction Stop + if ($report.schema -ne 'scim.validation-run-details' -or $report.collection.status -ne 'Completed' -or + $report.diagnosticScope.name -ne 'FailureFocused' -or -not $report.finalResults.testResults -or + $report.fullRun.status -notin @('Succeeded', 'Failed', 'Cancelled', 'TimedOut', 'Aborted')) { + throw 'Offline correction requires a completed FailureFocused report with recorded test results and a terminal root run.' + } + $requests = $report.runtime.evidenceRequests.ContentPayload + if (-not $requests -or $requests.count -lt 1 -or -not $requests.Contains('failed') -or $requests.failed -ne 0) { + throw 'Offline correction requires recorded content-request metrics proving zero failed downloads.' + } + $coreFailures = @($report.tests | Where-Object { $_.applicable -and -not $_.authenticationMethod -and -not $_.passing -and -not $_.nonBlockingWarning }) + $recordedFailure = $coreFailures.Count -gt 0 -or + [string]$report.finalResults.overallResult -match '^(?i:fail|failed)$' -or + $report.fullRun.status -ne 'Succeeded' -or + @($report.authentication.methodResults | Where-Object { -not $_.passing }).Count -gt 0 + if (-not $recordedFailure) { throw 'This narrow offline correction cannot establish gallery readiness; an independently recorded non-ready outcome is required.' } + $classificationReason = 'Successful data-producing action supplied no readable output evidence.' + $corrected = [System.Collections.Generic.List[object]]::new() + $resolvedSources = [System.Collections.Generic.HashSet[string]]::new() + $base = "https://management.azure.com/subscriptions/$([uri]::EscapeDataString($report.environment.subscriptionId.value))/resourceGroups/$([uri]::EscapeDataString($report.environment.resourceGroupName.value))/providers/Microsoft.Web/sites/$([uri]::EscapeDataString($report.environment.logicAppName.value))/hostruntime/runtime/webhooks/workflow/api/management" + function Resolve-RecordedComposeEmptyString($Entry, [string] $Uri, [string] $ActionName) { + $availability = $Entry.contentAvailability + $p = $Entry.evidence.properties + $source = "$Uri/outputs" + $matches = @($report.evidenceFailures | Where-Object { $_.required -and $_.source -eq $source -and $_.reason -eq $classificationReason }) + $protected = @($Entry.securityExclusions | Where-Object source -eq 'outputs').Count -gt 0 + if ($Entry.type -eq 'Compose' -and $Entry.status -eq 'Succeeded' -and $Entry.collectionState -eq 'Completed' -and + $Entry.payloadCollection -in @('TopLevelDiagnosticPayloads', 'RequiredDiagnosticRepetition') -and + $availability.outputs.status -eq 'Unavailable' -and $Entry.Contains('outputs') -and $null -eq $Entry.outputs -and + $availability.inputs.status -in @('Collected', 'ReusedSanitized') -and + $Entry.inputs -is [string] -and $Entry.inputs.Length -eq 0 -and + $p.outputsLink.uri -and $p.outputsLink.contentSize -eq 2 -and -not $protected -and $matches.Count -eq 1) { + $Entry.outputs = '' + $availability.outputs = @{ + status = 'Collected'; presenceSource = 'OfflineComposeIdentityCorrection' + reason = 'Legacy empty-string parsing bug corrected from collected Compose input identity, two-byte output metadata, matching classification-only gap and zero failed content downloads. No new HTTP observation is asserted.' + } + [void]$resolvedSources.Add($source) + $corrected.Add(@{ source = $source; action = $ActionName; repetition = $Entry.name; valueKind = 'EmptyString'; outputContentSize = 2 }) + } + $kind = if ($Entry.repetitionCollection.kind -in @('repetitions', 'scopeRepetitions')) { $Entry.repetitionCollection.kind } else { 'repetitions' } + foreach ($rep in $Entry.repetitions) { + Resolve-RecordedComposeEmptyString $rep "$Uri/$kind/$([uri]::EscapeDataString($rep.name))" $ActionName + } + } + foreach ($workflow in $report.workflows) { + $runUri = "$base/workflows/$([uri]::EscapeDataString($workflow.name))/runs/$([uri]::EscapeDataString($workflow.runId))" + foreach ($action in $workflow.actions) { + Resolve-RecordedComposeEmptyString $action "$runUri/actions/$([uri]::EscapeDataString($action.name))" $action.name + } + } + if (-not $corrected.Count) { throw 'No empty-string Compose classification gaps have sufficient recorded proof for offline correction.' } + $originalExitCode = $report.galleryReadiness.exitCode + $report.evidenceFailures = @($report.evidenceFailures | Where-Object { + -not ($_.required -and $_.reason -eq $classificationReason -and $resolvedSources.Contains([string]$_.source)) + }) + $missing = @($report.evidenceFailures | Where-Object required).Count + $report.completeness.requiredUnavailableCount = $missing + $report.completeness.complete = $missing -eq 0 + $report.galleryReadiness.ready = $false + $report.galleryReadiness.status = if ($missing) { 'Incomplete' } else { 'NotReady' } + $report.galleryReadiness.result = if ($missing) { 'INCOMPLETE' } else { 'NOT_GALLERY_READY' } + $report.galleryReadiness.exitCode = if ($missing) { 2 } else { 1 } + $report.galleryReadiness.reason = if ($missing) { 'Required evidence remains unavailable after narrow offline empty-string correction.' } else { 'Evidence is complete within the recorded diagnostic scope; original functional, authentication or runtime failures still prevent gallery readiness.' } + $report.offlineFinalization = @{ + kind = 'LegacyComposeEmptyStringPresenceCorrection'; correctedOutputCount = $corrected.Count + sourceReportSha256 = (Get-FileHash -LiteralPath $ReportPath -Algorithm SHA256).Hash.ToLowerInvariant() + corrector = @{ fileName = [IO.Path]::GetFileName($PSCommandPath); sha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant() } + originalExitCode = $originalExitCode; networkRequests = 0; corrections = @($corrected) + provenance = 'Original run/source provenance, semantic tests, request metrics and failed functional results are preserved. Source report was read only; this is not a new collection or deployment.' + } + return $report +} + +function Write-ScimValidationReport { + param($Report, [string] $WorkDirectory, [switch] $Zip) + [void][IO.Directory]::CreateDirectory($WorkDirectory) + $jsonPath = Join-Path $WorkDirectory 'validation-run-details.json' + $markdownPath = Join-Path $WorkDirectory 'validation-report.md' + [IO.File]::WriteAllText($jsonPath, (ConvertTo-ScimCanonicalJson $Report) + "`n", [Text.UTF8Encoding]::new($false)) + function Cell($value) { return ([System.Net.WebUtility]::HtmlEncode([string]$value) -replace '\|', '|' -replace '[\r\n]', ' ') } + $lines = [System.Collections.Generic.List[string]]::new() + $lines.Add('# SCIM production validation report') + $lines.Add('') + $lines.Add("**$($Report.galleryReadiness.status)** — $($Report.galleryReadiness.reason)") + $lines.Add('') + $lines.Add("Schema: $($Report.schemaVersion). Orchestrator run: $(if ($Report.orchestratorRunId) { Cell $Report.orchestratorRunId } else { 'Not executed' }).") + $lines.Add("Agent: $(Cell $Report.provenance.agent.fileName); SHA256: $(Cell $Report.provenance.agent.sha256). Source revision: $(Cell $Report.provenance.sourceGit.revision); dirty: $(Cell $Report.provenance.sourceGit.dirty).") + $lines.Add("Required unavailable evidence: $($Report.completeness.requiredUnavailableCount). Optional unavailable evidence: $($Report.completeness.optionalUnavailableCount).") + $lines.Add("Reporting source VERSION: $(Cell $Report.provenance.sourceVersion.value); VERSION SHA256: $(Cell $Report.provenance.sourceVersion.sha256). $(Cell $Report.provenance.sourceVersion.scope)") + $lines.Add("Historical deployed version: $(Cell $Report.provenance.executedVersion.value). Recorded authentication policy: $(Cell $Report.authentication.recordedPolicy). $(Cell $Report.historicalRun.unavailableReason)") + $lines.Add("Caller-recorded historical source VERSION: $(Cell $Report.historicalRun.metadata.sourceVersion.value); original VERSION SHA256: $(Cell $Report.historicalRun.metadata.sourceVersion.sha256). Historical policy claim: $(Cell $Report.historicalRun.metadata.authenticationPolicy).") + $lines.Add("Deployed version matches reporting source: $(Cell $Report.provenance.comparison.deployedVersionMatchesSource). $(Cell $Report.provenance.comparison.reason)") + $lines.Add("Intentional security exclusions: $($Report.completeness.securityExcludedCount). $(Cell $Report.completeness.scope)") + $lines.Add("Diagnostic scope: $(Cell $Report.diagnosticScope.name). $(Cell $Report.diagnosticScope.metadata) $(Cell $Report.diagnosticScope.payloads) $(Cell $Report.diagnosticScope.successfulRepeatedPollPayloads)") + if ($Report.offlineFinalization) { $lines.Add("Offline finalization: $(Cell $Report.offlineFinalization.kind); corrected outputs: $($Report.offlineFinalization.correctedOutputCount); source report SHA256: $(Cell $Report.offlineFinalization.sourceReportSha256). $(Cell $Report.offlineFinalization.provenance)") } + if ($Report.resume.requested) { $lines.Add("Resumed sanitized terminal payloads: $($Report.resume.reusedPayloadCount). $(Cell $Report.resume.policy)") } + $lines.Add("Full run: $(Cell $Report.fullRun.timing.startUtc) → $(Cell $Report.fullRun.timing.endUtc); duration: $(Cell $Report.fullRun.timing.durationMilliseconds) ms.") + $lines.Add('') + $lines.Add('## Agent and setup timing') + $lines.Add("Agent setup-to-cleanup window: $(Cell $Report.agentRun.timing.startUtc) → $(Cell $Report.agentRun.timing.endUtc); duration: $(Cell $Report.agentRun.timing.durationMilliseconds) ms; status: $(Cell $Report.agentRun.status); timing: $(Cell $Report.agentRun.timingState).") + $lines.Add("$(Cell $Report.agentRun.timing.unavailableReason) $(Cell $Report.agentRun.statusUnavailableReason)") + $lines.Add("Recorded events: $(Cell $Report.agentEvents.status). $(Cell $Report.agentEvents.unavailableReason)") + $lines.Add('| Event | Kind | Parent | Status | UTC start | UTC end | Duration (ms) | Timing availability | Details |') + $lines.Add('|---|---|---|---|---|---|---|---|---|') + foreach ($event in $Report.agentEvents.events) { + $detailText = if ($event.details -is [string]) { $event.details } elseif ($null -ne $event.details) { ConvertTo-ScimCanonicalJson $event.details } else { '' } + $lines.Add("| $(Cell $event.name) | $(Cell $event.kind) | $(Cell $event.parentName) | $(Cell $event.status) | $(Cell $event.timing.startUtc) | $(Cell $event.timing.endUtc) | $(Cell $event.timing.durationMilliseconds) | $(Cell $event.timingState): $(Cell $event.unavailableReason) | $(Cell $detailText) |") + } + $lines.Add('') + $lines.Add('Runtime success is not test success. The JSON contains sanitized action, scope, trigger, repetition, provenance, environment and timing details.') + $lines.Add('') + $lines.Add('## Workflow execution') + $lines.Add('| Workflow | Exact run ID | Status | UTC start | UTC end | Duration (ms) |') + $lines.Add('|---|---|---|---|---|---|') + foreach ($wf in $Report.workflows) { $lines.Add("| $(Cell $wf.name) | $(Cell $wf.runId) | $(Cell $wf.status) | $(Cell $wf.timing.startUtc) | $(Cell $wf.timing.endUtc) | $(Cell $wf.timing.durationMilliseconds) |") } + $lines.Add('') + $lines.Add('## Selected authentication methods') + $lines.Add((Cell ($Report.authentication.selectedMethods -join ', '))) + $lines.Add((Cell $Report.authentication.policy)) + $lines.Add("Selection source: $(Cell $Report.authentication.selectionSource). Recorded executed selection: $(Cell ($Report.authentication.recordedSelectedMethods -join ', ')). Selection mismatch: $(Cell $Report.authentication.selectionMismatch).") + $lines.Add("Recorded authentication aggregate: $(Cell $Report.authentication.recordedAggregateResult). Aggregate contradicts selected-method policy: $(Cell $Report.authentication.aggregatePolicyMismatch).") + foreach ($method in $Report.authentication.methodResults) { $lines.Add("- $(Cell $method.method): passing=$($method.passing); eligible=$($method.eligible). $(Cell $method.reason)") } + $lines.Add("Direct SCIM request transport: $(Cell $Report.authentication.scimRequestTransport.value). $(Cell $Report.authentication.scimRequestTransport.unavailableReason)") + $lines.Add('') + $lines.Add('## Semantic test results') + $lines.Add("Passed: $($Report.testSummary.passed); optional warnings: $($Report.testSummary.warnings); failed: $($Report.testSummary.failed); not applicable: $($Report.testSummary.notApplicable); unknown: $($Report.testSummary.unknown).") + $lines.Add('| Test | Result | Applicable | Reason |') + $lines.Add('|---|---|---|---|') + foreach ($test in $Report.tests) { $lines.Add("| $(Cell $test.name) | $(Cell $test.semanticResult) | $(Cell $test.applicable) | $(Cell $test.applicabilityReason) |") } + $lines.Add('') + $lines.Add('## Unavailable evidence and next actions') + foreach ($failure in $Report.evidenceFailures) { $lines.Add("- $(Cell $failure.source): $(Cell $failure.reason) Required: $($failure.required). Supply the missing evidence or resolve read permissions and export again.") } + foreach ($exclusion in $Report.securityExclusions) { $lines.Add("- Security exclusion: $(Cell $exclusion.source). $(Cell $exclusion.reason)") } + [IO.File]::WriteAllText($markdownPath, ($lines -join "`n") + "`n", [Text.UTF8Encoding]::new($false)) + if ($Zip) { Compress-Archive -LiteralPath @($jsonPath, $markdownPath) -DestinationPath (Join-Path $WorkDirectory 'validation-report.zip') -Force } +} + +if ($MyInvocation.InvocationName -ne '.') { + $ErrorActionPreference = 'Stop' + try { + if (-not $WorkDirectory -or -not $AgentPath) { throw 'WorkDirectory and AgentPath are required.' } + $WorkDirectory = [IO.Path]::GetFullPath($WorkDirectory) + $checkpointPath = Join-Path $WorkDirectory 'validation-run-details.checkpoint.json' + $previousReport = $null + if (-not $ResumeReportPath -and (Test-Path -LiteralPath $checkpointPath)) { + $candidate = Get-Content -LiteralPath $checkpointPath -Raw | ConvertFrom-Json -AsHashtable + if ($candidate.orchestratorRunId -eq $OrchestratorRunId -and $candidate.environment.subscriptionId.value -eq $SubscriptionId -and + $candidate.environment.resourceGroupName.value -eq $ResourceGroupName -and $candidate.environment.logicAppName.value -eq $LogicAppName) { $ResumeReportPath = $checkpointPath } + } + if ($ResumeReportPath) { + if (-not [IO.Path]::IsPathRooted($ResumeReportPath)) { $ResumeReportPath = Join-Path $WorkDirectory $ResumeReportPath } + $previousReport = Get-Content -LiteralPath $ResumeReportPath -Raw | ConvertFrom-Json -AsHashtable + } + if ($MetadataPath -and [IO.Path]::GetFileName($MetadataPath) -match '(?i)(parameters|appsettings).*\.json$') { throw 'Configuration files are not accepted as report metadata.' } + $metadata = if ($MetadataPath) { Get-Content -LiteralPath $MetadataPath -Raw | ConvertFrom-Json -AsHashtable } else { @{} } + if (-not $metadata.environment) { $metadata.environment = @{} } + if (-not $metadata.harness) { $metadata.harness = @{} } + if ($TenantId) { $metadata.environment.tenantId = $TenantId } + if ($metadata.provisioningLogPath -and -not [IO.Path]::IsPathRooted($metadata.provisioningLogPath)) { $metadata.provisioningLogPath = Join-Path $WorkDirectory $metadata.provisioningLogPath } + if (-not $WorkflowDirectory) { + $WorkflowDirectory = if (Test-Path -LiteralPath (Join-Path $WorkDirectory 'Orchestrator_Workflow.json')) { $WorkDirectory } else { $PSScriptRoot } + } + $runtime = @{ powerShell = $PSVersionTable.PSVersion.ToString(); os = [System.Runtime.InteropServices.RuntimeInformation]::OSDescription; azureCli = $null; azureCliUnavailableReason = $null } + $token = $null + $authenticationState = $null + $authFailure = $null + if ($OrchestratorRunId) { + try { + if (-not $SubscriptionId -or -not $ResourceGroupName -or -not $LogicAppName) { throw 'SubscriptionId, ResourceGroupName and LogicAppName are required for executed runs.' } + $tokenObject = Get-ScimArmAuthentication -SubscriptionId $SubscriptionId -TenantId $TenantId + $runtime.armAuthenticationSelector = $tokenObject.selector + $token = $tokenObject.accessToken + if (-not $token) { throw 'Azure CLI returned no ARM access token.' } + $KnownSecrets += $token + $authenticationState = New-ScimArmRequestState -Authentication $tokenObject -SubscriptionId $SubscriptionId -TenantId $TenantId + if (-not $TenantId) { $metadata.environment.tenantId = $tokenObject.tenant } + $account = & az account show --subscription $SubscriptionId --query '{tenantId:tenantId,operatorIdentity:user}' --output json --only-show-errors 2>$null + if ($LASTEXITCODE -eq 0) { + $accountObject = $account | ConvertFrom-Json -AsHashtable + if (-not $TenantId -or $accountObject.tenantId -eq $TenantId) { $metadata.environment.operatorIdentity = $accountObject.operatorIdentity } + } + $version = & az version --query '"azure-cli"' --output tsv 2>$null + if ($LASTEXITCODE -eq 0) { $runtime.azureCli = [string]$version } else { $runtime.azureCliUnavailableReason = 'Azure CLI version could not be read.' } + } catch { $authFailure = $_.Exception.Message; $runtime.azureCliUnavailableReason = 'Azure CLI authentication or environment discovery failed.' } + } else { $runtime.azureCliUnavailableReason = 'No execution requested; Azure CLI was not invoked.' } + $refreshingRequest = ${function:Invoke-ScimRefreshingRequest} + $request = { + param($uri, $arm, $receipt) + if ($authFailure) { throw $authFailure } + return ,(& $refreshingRequest $authenticationState $uri $arm $receipt) + }.GetNewClosure() + $report = New-ScimValidationReport -SubscriptionId $SubscriptionId -ResourceGroupName $ResourceGroupName -LogicAppName $LogicAppName ` + -OrchestratorRunId $OrchestratorRunId -AgentPath $AgentPath -WorkflowDirectory $WorkflowDirectory -Metadata $metadata ` + -KnownSecrets $KnownSecrets -Request $request -RuntimeMetadata $runtime -WorkDirectory $WorkDirectory -AuthenticationState $authenticationState -PreviousReport $previousReport -CheckpointPath $checkpointPath + Write-ScimValidationReport $report $WorkDirectory -Zip:$Zip + if (Test-Path -LiteralPath $checkpointPath) { Remove-Item -LiteralPath $checkpointPath -Force } + $token = $null; $tokenResponse = $null; $tokenObject = $null; $KnownSecrets = @() + if ($authenticationState) { $authenticationState.authentication = $null; $authenticationState.secretLiterals.Clear() } + Write-Host "Sanitized report written. Readiness: $($report.galleryReadiness.status). Required evidence unavailable: $($report.completeness.requiredUnavailableCount)." + exit $report.galleryReadiness.exitCode + } catch { + # Never echo raw exception bodies, CLI output, metadata, headers or credentials. + Write-Error 'Report export failed before completion. Saved sanitized actions remain in validation-run-details.checkpoint.json when available; rerun with the same root and site to resume. No gallery readiness is claimed; existing canonical reports were not refreshed.' -ErrorAction Continue + exit 2 + } +} diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Orchestrator_Parameters.json b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Orchestrator_Parameters.json index fe6c9c76..41d60441 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Orchestrator_Parameters.json +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Orchestrator_Parameters.json @@ -58,7 +58,7 @@ "description": "Overall bundle and agent version, copied from the same-bundle VERSION file." }, "type": "String", - "value": "14.0" + "value": "15.0" }, "attributeTransforms": { "metadata": { diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SCIM-Validation-Test-Overview.md b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SCIM-Validation-Test-Overview.md index c8c21a0b..80b203c5 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SCIM-Validation-Test-Overview.md +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SCIM-Validation-Test-Overview.md @@ -177,17 +177,29 @@ This applies to: Create_Group, Update_Group, Group_Update_Add_Member, Group_Upda | Group tests `FAILED` (no Group schema) | **Failed** | Must add Group support to SCIM endpoint | | Any mandatory test `FAILED` | **Failed** | Must fix failing test(s) before submission | -## Authentication regression checks +## Shareable evidence and regression checks -Run `.\Test-Authentication.ps1` after changing authentication behavior. -The suite evaluates the actual workflow expressions and direct-request -token-cache wiring. Live federation still requires a federation-capable -environment; offline exchange parity is not a live authentication result. +The [canonical agent](scim-onboarding.agent.md) creates a version/timestamp/local-ID subfolder under its starting current directory before validation. It records VERSION and source hashes, selected methods, and setup-step timings. Its report step uses [Export-ScimValidationReport.ps1](Export-ScimValidationReport.ps1) to produce `validation-run-details.json`, `validation-report.md`, and an optional portable archive for the **exact** run ID, plus versioned shareable copies. -See [Authentication-Changes.md](Authentication-Changes.md) for the design -and the boundary between authentication readiness and functional readiness. +Evidence includes UTC workflow/test/action/repetition timings, nested failures and retries, sanitized HTTP requests/responses, source/deployment provenance, and environment/operator identifiers. Protected token payloads remain excluded. Unavailable runtime/model information and missing evidence are explicitly reported, never guessed. Review the artifacts before sharing: redaction removes credentials, not all organizational identifiers or test-user data. + +Generate the final report after the exact Orchestrator run and its child workflows have reached terminal states. A report collected while the run is active is an incomplete snapshot, not a final readiness result. Detailed evidence collection can take longer than a short Azure CLI access-token lifetime; collection failures must remain visible until the missing evidence has been retrieved successfully. + +To resume an interrupted collection, stop or verify completion of the previous process and rerun the same exact-run invocation without `-ResumeReportPath`; the matching sanitized checkpoint is loaded automatically. To import a prior final report explicitly, add `-ResumeReportPath "$WorkDirectory\validation-run-details.json"`. The exporter reuses successful sanitized terminal payloads, refreshes metadata, and retries missing evidence. Run only one collector per work directory. Resuming does not turn missing evidence into a pass. Until-loop history uses the container timing/counts and supported nested action repetitions rather than an unsupported loop repetition endpoint. + +Run the dependency-free PowerShell regressions after changes: + +```powershell +.\Test-Authentication.ps1 +.\Test-ScimValidationReport.ps1 +``` + +The authentication suite evaluates the actual workflow expressions, including OAuth-only, WIF-only, alternative-method failures, legacy inference, non-authentication failures, and every direct SCIM request's cache/transport wiring. Live WIF testing still requires a WIF-capable environment; offline exchange parity is not a live authentication result. + +For reporting scope, provenance, data transformations, and root-cause analysis, see [Validation reporting changes](Validation-Reporting-Changes.md). The prerequisite authentication behavior is documented separately in [Authentication changes](Authentication-Changes.md). --- + ## How to Run ### Method A: AI Agent (Automated, 30–60 min) diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SetupLogicApp-Standard-Agent.md b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SetupLogicApp-Standard-Agent.md index 515e5882..e68ce7ed 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SetupLogicApp-Standard-Agent.md +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/SetupLogicApp-Standard-Agent.md @@ -434,19 +434,39 @@ The agent clears undeclared credentials and reports those tests as declared method must be supplied or reported as blocked; they are not silently treated as an unsupported method. -### Authentication readiness +### Authentication readiness and run evidence Every selected authentication method must pass its own test (AND). -Selecting both OAuth and WIF requires both to pass; a selected failure, -skip, or missing result blocks authentication readiness. An undeclared -method is Not applicable and issues no authentication requests. -Other mandatory functional failures remain independent readiness gates. - -Use one coherent bundle: seven workflows, the complete parameters file, -the canonical agent, authentication tests, and [VERSION](VERSION). -Keep `LogicAppVersion` synchronized with VERSION. This change does not -require a portable report exporter; reporting is a separate follow-up. -See [Authentication-Changes.md](Authentication-Changes.md). +Selecting both OAuth and WIF requires both to pass; one passing method +cannot hide a selected failure, skipped result, or missing result. +Other mandatory functional failures still prevent readiness. +For example, a WIF-only integration with a passing federation test and +no OAuth endpoint is authentication-ready; OAuth is Not applicable. + +Use one coherent agent/template bundle: seven workflows, the complete +parameters file, `VERSION`, and the report exporter from the same source revision. +Do not replace locally updated templates with older downloads. + +The [VERSION](VERSION) file is the sole overall and agent version. +Bump it for each bundle change and keep `LogicAppVersion` synchronized. +Before asking for inputs, the agent creates a unique subfolder under +its starting current directory, for example +`scimrun-v15.0-20261016T170000Z-a1b2c3d4`. The short name includes the +version, UTC start time, and local run ID; metadata records full Azure +identifiers. Final shareable filenames also include the exact Azure +run ID. Stable helper filenames remain available within that folder. + +For every outcome, including setup blockers, the agent exports a +sanitized **`validation-run-details.json`** and human-readable report in +the run work directory, with an optional portable archive. These include +the exact run ID, source/deployed version, source/deployment hashes, selected authentication +methods, timings, environment/operator identifiers, failure diagnostics, +and explicit evidence gaps. Token payloads remain protected. Unexposed +model, reasoning, and context settings are recorded as unavailable. +Review the artifacts before sharing because organizational identifiers +and test-user data may remain. See the +[canonical agent's report step](scim-onboarding.agent.md#step-6g-export-a-shareable-run-report-required-for-every-outcome). + ### Question 4: Azure Subscription Selection > “Which Azure subscription would you like to use?” diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-Authentication.ps1 b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-Authentication.ps1 index cee6b399..d2ec3090 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-Authentication.ps1 +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-Authentication.ps1 @@ -282,6 +282,28 @@ Assert-Equal $authenticationTests.Count 2 'The existing report inventory retains foreach ($test in $authenticationTests) { Assert-Equal $test.testCategory 'Declared/StrictSuccess' 'Each declared authentication test is individually strict' } +$agentText = Get-Content -LiteralPath "$root\scim-onboarding.agent.md" -Raw +$startup = [regex]::Match($agentText, '(?s)```powershell\r?\n(?.*?)\r?\n```') +Assert-Equal $startup.Success $true 'Canonical agent contains executable startup-folder instructions' +$testDirectory = Join-Path ([IO.Path]::GetTempPath()) ('scim-startup-' + [guid]::NewGuid().ToString('N')) +$null = New-Item -ItemType Directory -Path $testDirectory -ErrorAction Stop +$createdDirectory = $null +Push-Location -LiteralPath $testDirectory +try { + $createdDirectory = & { + $sourceWorkflowDirectory = $root + & ([scriptblock]::Create($startup.Groups['code'].Value + "`n`$workDirectory")) + } + Assert-Equal (Split-Path -Parent $createdDirectory) $testDirectory 'Run folder is a direct child of the starting current directory' + $version = (Get-Content -LiteralPath "$root\VERSION" -Raw).Trim() + Assert-Equal ((Split-Path -Leaf $createdDirectory) -match ('^scimrun-v' + [regex]::Escape($version) + '-\d{8}T\d{6}Z-[a-f0-9]{8}$')) $true 'Run folder contains the actual VERSION, UTC start and unique local run ID' +} finally { + Pop-Location + if ($createdDirectory -and (Test-Path -LiteralPath $createdDirectory)) { + Remove-Item -LiteralPath $createdDirectory -ErrorAction Stop + } + Remove-Item -LiteralPath $testDirectory -ErrorAction Stop +} if (-not $orchestrator.definition.actions.Evaluate_Test_Results.inputs.overallLogicAppResult.Contains("not(equals(coalesce(body('Get_SCIMTests_Output_Content')?['authenticationModelResult'],'FAILED'),'PASSED'))")) { throw 'Orchestrator must reject missing or non-passing authentication readiness.' } diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 new file mode 100644 index 00000000..5207568d --- /dev/null +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 @@ -0,0 +1,1000 @@ +#requires -Version 7.0 +<# +.SYNOPSIS +Runs dependency-free, offline reporting regression tests. No Azure CLI or network. +.EXAMPLE +pwsh -NoProfile -File .\Test-ScimValidationReport.ps1 +.EXAMPLE +pwsh -NoProfile -File .\Test-ScimValidationReport.ps1 -EvidenceDirectory .\report-contract-evidence +#> +[CmdletBinding()] +param([string] $EvidenceDirectory) +$ErrorActionPreference = 'Stop' +. (Join-Path $PSScriptRoot 'Export-ScimValidationReport.ps1') +$script:checks = 0 +function Assert-Report($Condition, [string] $Message) { + $script:checks++ + if (-not $Condition) { throw "Assertion failed: $Message" } +} + +$fixtureDirectory = Join-Path $PSScriptRoot ('.scim-report-tests-' + [guid]::NewGuid().ToString('N')) +[void][IO.Directory]::CreateDirectory($fixtureDirectory) +try { + $secret = 'fixture-opaque-secret+/=42' + $jwt = 'eyJhbGciOiJub25lIn0.eyJzdWIiOiJmaXh0dXJlIn0.signature' + $fixtures = @( + @{ headers = @{ Authorization = 'Bearer raw-auth-token'; 'Proxy-Authorization' = 'Basic raw-proxy-token'; Cookie = 'session=raw-cookie'; 'Set-Cookie' = 'session=raw-set-cookie'; 'X-Api-Key' = 'raw-api-key'; 'Content-Type' = 'application/json'; 'x-ms-request-id' = 'safe-correlation'; 'Custom-Credential' = 'raw-custom-header' } }, + @{ embedded = '{"inner":"{\"client_secret\":\"raw-embedded-secret\"}"}' }, + @{ body = 'client_secret=raw-form-secret&client_assertion=raw-form-assertion&access_token=raw-form-access&refresh_token=raw-form-refresh&id_token=raw-form-id&password=raw-form-password' }, + @{ body = 'client_secret%3Draw-encoded-secret%26grant_type%3Dclient_credentials' }, + @{ error = 'Failure access_token: raw-error-token; client_secret="raw-error-secret" password=raw-error-password' }, + @{ uri = 'https://sample.blob.core.windows.net/container/data?sv=2026&sig=raw-sas-signature&custom=raw-query-secret#raw-fragment-secret' }, + @{ nested = @(@{ token = 'raw-nested-token'; connectionString = 'raw-connection-string'; result = 'PASSED' }); jwt = $jwt; literal = $secret }, + @{ body = '{"access_token":"raw-payload-token","opaque":"raw-token-response-extra"}' }, + @{ headersText = "Authorization: Bearer raw-text-auth`nCookie: session=raw-text-cookie`nSet-Cookie: raw-text-setcookie" }, + @{ value = 'DefaultEndpointsProtocol=https;AccountName=sample;AccountKey=raw-account-key;EndpointSuffix=core.windows.net' }, + @{ value = ('prefix ' + [uri]::EscapeDataString($secret) + ' suffix') }, + @{ value = 'https://sample.azure.com/path?api_key=raw-url-key&password=raw-url-password' }, + @{ value = '{"error":"{\"refresh_token\":\"raw-double-json-token\"}"}' }, + @{ error = 'Request failed with {\"access_token\":\"raw-escaped-error-token\"}' }, + @{ error = 'access token is raw-spaced-token; client secret: raw-spaced-secret' }, + @{ error = 'client_secret=raw-html-secret' }, + @{ error = 'client\u005fsecret=raw-unicode-secret' } + ) + $safe = Protect-ScimValue $fixtures @($secret) + $safeJson = ConvertTo-ScimCanonicalJson $safe + foreach ($forbidden in @( + 'raw-auth-token', 'raw-proxy-token', 'raw-cookie', 'raw-set-cookie', 'raw-api-key', 'raw-custom-header', + 'raw-embedded-secret', 'raw-form-secret', 'raw-form-assertion', 'raw-form-access', 'raw-form-refresh', + 'raw-form-id', 'raw-form-password', 'raw-encoded-secret', 'raw-error-token', 'raw-error-secret', 'raw-error-password', + 'raw-sas-signature', 'raw-query-secret', 'raw-fragment-secret', 'raw-nested-token', 'raw-connection-string', + 'raw-payload-token', 'raw-token-response-extra', 'raw-text-auth', 'raw-text-cookie', 'raw-text-setcookie', + 'raw-account-key', 'raw-url-key', 'raw-url-password', 'raw-double-json-token', + 'raw-escaped-error-token', 'raw-spaced-token', 'raw-spaced-secret', 'raw-html-secret', 'raw-unicode-secret', $secret, $jwt + )) { Assert-Report (-not $safeJson.Contains($forbidden)) "Secret redacted: $forbidden" } + Assert-Report ($safeJson.Contains('safe-correlation')) 'Safe correlation header retained' + Assert-Report ($safeJson.Contains('application/json')) 'Allowlisted content-type retained' + Assert-Report ($safeJson.Contains('PASSED')) 'Semantic result retained' + Assert-Report ((ConvertTo-ScimCanonicalJson (Protect-ScimValue $safe @($secret))) -eq $safeJson) 'Repeated sanitization preserves already-redacted fixture content' + $prose = 'Acquire an ARM token for the selected tenant.' + Assert-Report ((Protect-ScimText $prose) -eq $prose) 'Harmless token-for prose is not treated as a credential assignment' + $maskedUrl = 'https://sample.azure.com/path?[REDACTED QUERY]' + Assert-Report ((Protect-ScimText (Protect-ScimText $maskedUrl)) -eq $maskedUrl) 'Query redaction marker is idempotent and does not accumulate suffixes' + Assert-Report ((Protect-ScimText 'access_token: [REDACTED]') -eq 'access_token: [REDACTED]') 'Credential-assignment redaction marker is idempotent' + foreach ($diagnosticText in @( + 'StatusCode: 400', 'statusCode=400', '"statusCode":400', 'STATUSCODE is 400', + '"errorCode":400', 'errorCode: SystemForCrossDomainIdentityManagementServiceIncompatible', + 'status-code=400; error_code=invalidSyntax', + 'StatusCode: 400; response: {"errorCode":400,"scimType":"invalidSyntax","detail":"Invalid filter."}' + )) { + Assert-Report ((Protect-ScimText $diagnosticText) -eq $diagnosticText) "Diagnostic code label is not an OAuth code assignment: $diagnosticText" + } + foreach ($codeAssignment in @( + 'code=fixture-oauth-code', 'code: fixture-oauth-code', 'code is fixture-oauth-code', + '"code":"fixture-oauth-code"', "'code'='fixture-oauth-code'", + 'authorization_code=fixture-oauth-code', '"authorization_code":"fixture-oauth-code"', + 'authorization-code: fixture-oauth-code', 'authorization code is fixture-oauth-code', + 'CODE=fixture-oauth-code' + )) { + $redactedCode = Protect-ScimText $codeAssignment + Assert-Report ($redactedCode.Contains('[REDACTED]') -and -not $redactedCode.Contains('fixture-oauth-code')) "Exact OAuth code assignment remains redacted: $codeAssignment" + Assert-Report ((Protect-ScimText $redactedCode) -eq $redactedCode) "OAuth code redaction stays idempotent: $codeAssignment" + } + $diagnosticReason = 'StatusCode: 400; response: {"errorCode":400,"scimType":"invalidSyntax"}; code=fixture-oauth-code; authorization_code=fixture-other-code' + $diagnosticLogs = Protect-ScimValue @{ provisioningLogs = @{ + errorCode = 'SystemForCrossDomainIdentityManagementServiceIncompatible'; reason = $diagnosticReason + } } + Assert-Report ($diagnosticLogs.provisioningLogs.reason.Contains('StatusCode: 400') -and + $diagnosticLogs.provisioningLogs.reason.Contains('"errorCode":400') -and + $diagnosticLogs.provisioningLogs.reason.Contains('"scimType":"invalidSyntax"') -and + $diagnosticLogs.provisioningLogs.errorCode -eq 'SystemForCrossDomainIdentityManagementServiceIncompatible') 'Nested provisioning diagnostics retain HTTP400, invalidSyntax and structured errorCode' + Assert-Report (-not $diagnosticLogs.provisioningLogs.reason.Contains('fixture-oauth-code') -and + -not $diagnosticLogs.provisioningLogs.reason.Contains('fixture-other-code')) 'OAuth code assignments are still redacted beside retained provisioning diagnostics' + Assert-Report ((Protect-ScimValue @{ message = 'ordinary failure'; result = 'FAILED' }).message -eq 'ordinary failure') 'Ordinary error retained' + Assert-Report ((ConvertTo-ScimCanonicalJson @{ z = 2; a = @{ b = 3; a = 1 } }) -eq (ConvertTo-ScimCanonicalJson @{ a = @{ a = 1; b = 3 }; z = 2 })) 'Canonical key ordering' + $authCalls = [System.Collections.Generic.List[object]]::new() + $fakeCli = { + param($commandArguments) + $authCalls.Add(@($commandArguments)) + if ('--tenant' -in $commandArguments -and '--subscription' -in $commandArguments) { + return @{ exitCode = 1; output = 'Please specify only one of subscription and tenant, not both' } + } + return @{ exitCode = 0; output = '{"accessToken":"fixture-arm-token","tenant":"fixture-tenant"}' } + }.GetNewClosure() + $tenantAuth = Get-ScimArmAuthentication -SubscriptionId 'fixture-subscription' -TenantId 'fixture-tenant' -InvokeCli $fakeCli + Assert-Report ($tenantAuth.selector -eq 'tenant' -and $tenantAuth.accessToken -eq 'fixture-arm-token') 'Explicit tenant authentication succeeds with CLI seam' + Assert-Report ('--tenant' -in $authCalls[0] -and '--subscription' -notin $authCalls[0]) 'Explicit tenant token request never combines mutually exclusive CLI selectors' + Assert-Report ('https://management.azure.com/' -in $authCalls[0]) 'Authentication targets the ARM resource' + $subscriptionAuth = Get-ScimArmAuthentication -SubscriptionId 'fixture-subscription' -InvokeCli $fakeCli + Assert-Report ($subscriptionAuth.selector -eq 'subscription' -and '--subscription' -in $authCalls[1] -and '--tenant' -notin $authCalls[1]) 'Subscription-only token acquisition remains supported' + $invalidAuthReason = $null + try { [void](Get-ScimArmAuthentication -TenantId 'fixture-tenant' -InvokeCli { @{ exitCode = 0; output = 'raw-invalid-auth-payload' } }) } + catch { $invalidAuthReason = $_.Exception.Message } + Assert-Report ($invalidAuthReason -and -not $invalidAuthReason.Contains('raw-invalid-auth-payload')) 'Malformed CLI output is not echoed into authentication diagnostics' + $wrongTenantReason = $null + try { [void](Get-ScimArmAuthentication -TenantId 'different-tenant' -InvokeCli $fakeCli) } + catch { $wrongTenantReason = $_.Exception.Message } + Assert-Report ($wrongTenantReason -match 'different tenant') 'Authentication cannot silently use a different tenant' + $failedAuthReason = $null + try { [void](Get-ScimArmAuthentication -TenantId 'fixture-tenant' -InvokeCli { @{ exitCode = 1; output = 'accessToken=raw-failed-auth-output' } }) } + catch { $failedAuthReason = $_.Exception.Message } + Assert-Report ($failedAuthReason -match 'CLI exit 1' -and -not $failedAuthReason.Contains('raw-failed-auth-output')) 'Failed authentication records a safe exit diagnostic, not raw output' + $expiryAuth = Get-ScimArmAuthentication -TenantId 'fixture-tenant' -InvokeCli { @{ exitCode = 0; output = '{"accessToken":"fixture-expiring-token","tenant":"fixture-tenant","expires_on":1767229200}' } } + Assert-Report ($expiryAuth.expiresUtc.ToUnixTimeSeconds() -eq 1767229200) 'CLI expires_on is retained as an absolute UTC expiry' + $transportFixture = @{ now = [datetimeoffset]'2026-01-01T00:00:00Z'; acquireCount = 0; sendCount = 0; fail401 = 0; status = 200; linkHadAuthorization = $false } + $clock = { $transportFixture.now }.GetNewClosure() + $acquire = { + param($subscription, $tenant) + $transportFixture.acquireCount++ + @{ accessToken = "refreshed-fixture-$($transportFixture.acquireCount)"; tenant = $tenant; expiresUtc = $transportFixture.now.AddHours(1) } + }.GetNewClosure() + $send = { + param($uri, $headers) + $transportFixture.sendCount++ + if ($uri -match 'fixture.azure.com') { $transportFixture.linkHadAuthorization = $headers.ContainsKey('Authorization') } + $status = if ($transportFixture.fail401 -gt 0) { $transportFixture.fail401--; 401 } else { $transportFixture.status } + if ($status -ne 200) { + $errorObject = [System.Net.Http.HttpRequestException]::new("Fixture final HTTP$status") + $errorObject.Data['StatusCode'] = $status + throw $errorObject + } + return @{ value = @() } + }.GetNewClosure() + $transport = New-ScimArmRequestState @{ accessToken = 'initial-fixture'; expiresUtc = $transportFixture.now.AddMinutes(4) } 'fixture-subscription' 'fixture-tenant' $acquire $send $clock + [void](Invoke-ScimRefreshingRequest $transport 'https://management.azure.com/expiry' $true) + Assert-Report ($transport.refreshCount -eq 1 -and $transportFixture.sendCount -eq 1) 'Five-minute expiry margin refreshes before sending the ARM request' + $transportFixture.fail401 = 1 + $before = $transportFixture.sendCount + [void](Invoke-ScimRefreshingRequest $transport 'https://management.azure.com/401' $true) + Assert-Report ($transport.retry401Count -eq 1 -and $transportFixture.sendCount -eq $before + 2) 'First ARM 401 refreshes credentials and retries exactly once' + Assert-Report ((Protect-ScimText 'refreshed-fixture-2' $transport.secretLiterals) -eq '[REDACTED]') 'Refreshed token literals remain available for defense-in-depth redaction' + $transportFixture.fail401 = 2 + $before = $transportFixture.sendCount + $final401 = $null + try { [void](Invoke-ScimRefreshingRequest $transport 'https://management.azure.com/still401' $true) } catch { $final401 = $_.Exception.Message } + Assert-Report ($transportFixture.sendCount -eq $before + 2 -and $final401 -match 'HTTP401') 'Repeated 401 preserves the final error after only one retry' + $transportFixture.status = 403 + $before = $transportFixture.sendCount + try { [void](Invoke-ScimRefreshingRequest $transport 'https://management.azure.com/403' $true) } catch { } + Assert-Report ($transportFixture.sendCount -eq $before + 1) 'Non-401 HTTP errors are not retried or hidden' + $transportFixture.status = 200 + [void](Invoke-ScimRefreshingRequest $transport 'https://fixture.azure.com/content' $false) + Assert-Report (-not $transportFixture.linkHadAuthorization) 'Signed content requests never receive ARM Authorization' + $sameCachedAuth = @{ accessToken = 'same-cached-token'; expiresUtc = $transportFixture.now.AddMinutes(3) } + $sameCached = New-ScimArmRequestState $sameCachedAuth 'fixture-subscription' 'fixture-tenant' { $sameCachedAuth }.GetNewClosure() $send $clock + [void](Invoke-ScimRefreshingRequest $sameCached 'https://management.azure.com/one' $true) + [void](Invoke-ScimRefreshingRequest $sameCached 'https://management.azure.com/two' $true) + Assert-Report ($sameCached.refreshCount -eq 1) 'Same near-expiry CLI cache token does not cause per-request refresh storms' + Assert-Report ($sameCached.secretLiterals.Count -eq 1) 'Repeated cached credentials do not duplicate secret-redaction literals' + $transportFixture.fail401 = 2 + $refreshingRequest = ${function:Invoke-ScimRefreshingRequest} + $finalErrorContext = @{ + failures = [System.Collections.Generic.List[object]]::new(); secrets = $transport.secretLiterals + request = { param($uri, $arm) & $refreshingRequest $transport $uri $arm }.GetNewClosure() + } + [void](Invoke-ScimEvidenceRequest $finalErrorContext 'https://management.azure.com/final401') + Assert-Report ($finalErrorContext.failures.Count -eq 1 -and $finalErrorContext.failures[0].httpStatus -eq 401) 'Exhausted 401 recovery records one final failure with its real HTTP status' + Assert-Report ($finalErrorContext.failures[0].reason -match 'HTTP401') 'Nested-host closure calls the captured refreshing transport rather than failing private function lookup' + $contractTests = @(@{ testName = 'Validate_Credentials_Test'; testCategory = 'Mandatory/StrictOneSuccess'; testResult = 'success' }) + for ($i = 1; $i -le 17; $i++) { $contractTests += @{ testName = "Core_$i"; testCategory = 'Mandatory/StrictSuccess'; testResult = 'success' } } + $contractTests += @{ testName = 'Restore_User_Test'; testCategory = 'Mandatory/StrictSuccess'; testResult = 'success (SoftDeleteRestore)' } + foreach ($name in @('Delete_Group_Test', 'Restore_Group_Test')) { $contractTests += @{ testName = $name; testCategory = 'Skippable/OptionalSuccess'; testResult = 'WARNING - optional provisioning evidence unavailable' } } + foreach ($name in @('Group_Update_Add_Member_Test', 'Group_Update_Remove_Member_Test', 'User_Update_Manager_Test')) { $contractTests += @{ testName = $name; testCategory = 'Skippable/StrictSuccess'; testResult = 'FAILED - fixture validation failure' } } + $contractTests += @{ testName = 'Federated_Identity_Test'; testCategory = 'Mandatory/StrictOneSuccess'; testResult = 'NOT_APPLICABLE - authentication method not declared for this validation' } + $classifiedContract = @(Get-ScimSemanticTests @{ testResults = $contractTests } @{} @() @('OAuth2ClientCredentials')) + Assert-Report ($classifiedContract.Count -eq 25 -and @($classifiedContract | Where-Object classification -eq 'Passed').Count -eq 19) 'Actual 25-test contract accepts ordinary and annotated success' + Assert-Report (@($classifiedContract | Where-Object classification -eq 'Warning').Count -eq 2 -and @($classifiedContract | Where-Object classification -eq 'Failed').Count -eq 3) 'OptionalSuccess warnings stay visible without becoming failed tests' + Assert-Report (@($classifiedContract | Where-Object classification -eq 'NotApplicable').Count -eq 1) 'Undeclared federation alternative is explicitly not applicable' + Assert-Report (@($classifiedContract | Where-Object { $_.authenticationMethod -eq 'OAuth2ClientCredentials' -and $_.passing }).Count -eq 1) 'Validate_Credentials_Test is recognized as the OAuth gallery alternative' + $blockingContract = @($classifiedContract | Where-Object { $_.applicable -and -not $_.passing -and -not $_.nonBlockingWarning -and -not $_.authenticationMethod }) + Assert-Report (($blockingContract.name -join ',') -eq 'Group_Update_Add_Member_Test,Group_Update_Remove_Member_Test,User_Update_Manager_Test') 'Only the three genuine core failures block the actual contract' + + $timing = Get-ScimTiming @{ startTime = '2026-01-01T01:00:00+01:00'; endTime = '2026-01-01T00:00:02Z' } + Assert-Report ($timing.durationMilliseconds -eq 2000) 'UTC elapsed time' + Assert-Report ($timing.startUtc -eq '2026-01-01T00:00:00.0000000Z') 'UTC start conversion' + Assert-Report ($null -eq (Get-ScimTiming @{}).durationMilliseconds) 'Missing timing is not invented' + Assert-Report ($null -ne (Get-ScimTiming @{ startTime = 'bad'; endTime = 'bad' }).unavailableReason) 'Invalid timing has reason' + + $paginationCalls = [System.Collections.Generic.List[string]]::new() + $ctx = @{ failures = [System.Collections.Generic.List[object]]::new(); secrets = @(); request = { + param($uri, $arm) + $paginationCalls.Add($uri) + if ($uri -match 'second') { return @{ value = @(@{ name = 'two' }) } } + return @{ value = @(@{ name = 'one' }); nextLink = 'https://management.azure.com/second' } + }.GetNewClosure() } + $pages = Get-ScimEvidencePages $ctx 'https://management.azure.com/first' + Assert-Report ($pages.Count -eq 2 -and $paginationCalls.Count -eq 2) 'Pagination collects every page' + foreach ($count in @(0, 1, 2)) { + $rawItems = @() + for ($i = 0; $i -lt $count; $i++) { $rawItems += [pscustomobject]@{ name = "raw-item-$i" } } + $rawContext = @{ + failures = [System.Collections.Generic.List[object]]::new(); secrets = @() + request = { param($uri, $arm) return ,$rawItems }.GetNewClosure() + } + $rawCollection = Get-ScimEvidencePages $rawContext 'https://management.azure.com/workflows' + Assert-Report ($rawCollection.Count -eq $count -and $rawContext.failures.Count -eq 0) "Raw collection arrays preserve $count items without an envelope" + } + $ctx.request = { param($uri, $arm) @{ value = @(); nextLink = 'https://evil.example/steal' } } + [void](Get-ScimEvidencePages $ctx 'https://management.azure.com/first') + Assert-Report ($ctx.failures.Count -eq 1) 'Pagination host escape recorded' + $ctx.failures.Clear() + $ctx.request = { param($uri, $arm) @{ value = @(); nextLink = $uri } } + [void](Get-ScimEvidencePages $ctx 'https://management.azure.com/first') + Assert-Report ($ctx.failures.Count -eq 1) 'Pagination cycle recorded' + $ctx.failures.Clear() + $ctx.request = { throw 'Denied access_token=raw-denied-token' } + [void](Get-ScimEvidencePages $ctx 'https://management.azure.com/denied?sig=raw-denied-sig') + Assert-Report (-not (ConvertTo-ScimCanonicalJson @($ctx.failures)).Contains('raw-denied')) 'Denied evidence reason and source sanitized' + $ctx.failures.Clear() + $unprotected = Get-ScimActionEvidence $ctx @{ name = 'Http'; properties = @{ + status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + } } 'https://management.azure.com/action' $false @() 'Http' + Assert-Report ($ctx.failures.Count -eq 1 -and $unprotected.contentAvailability.outputs.status -eq 'Unavailable') 'Missing HTTP output is not silently complete' + $valueCases = @( + @{ name = 'Null'; value = $null }, @{ name = 'EmptyString'; value = '' }, + @{ name = 'False'; value = $false }, @{ name = 'Zero'; value = 0 }, + @{ name = 'EmptyArray'; value = @() }, @{ name = 'EmptyObject'; value = @{} }, + @{ name = 'NullString'; value = 'null' }, @{ name = 'FalseString'; value = 'false' }, + @{ name = 'ZeroString'; value = '0' }, @{ name = 'WhitespaceString'; value = ' ' } + ) + foreach ($valueCase in $valueCases) { + $valueContext = @{ secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); cachedActions = @{} + request = { throw 'Inline value unexpectedly requested network access.' } } + $valueAction = @{ name = 'Compose_Value'; properties = @{ + status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + inputs = $valueCase.value; outputs = $valueCase.value + } } + $valueEvidence = Get-ScimActionEvidence $valueContext $valueAction 'https://management.azure.com/value' $false @() 'Compose' + Assert-Report ($valueContext.failures.Count -eq 0 -and $valueEvidence.contentAvailability.outputs.status -eq 'Collected' -and $valueEvidence.contentAvailability.inputs.status -eq 'Collected') "Explicit inline $($valueCase.name) is present evidence, not an absent payload" + Assert-Report ((ConvertTo-ScimCanonicalJson @{ value = $valueEvidence.outputs }) -eq (ConvertTo-ScimCanonicalJson @{ value = $valueCase.value })) "Inline $($valueCase.name) retains its exact JSON value" + $valueContext.request = { + param($uri, $arm, $receipt) + $receipt.httpStatus = 200 + return ,$valueCase.value + }.GetNewClosure() + $valueAction.properties.Remove('outputs') + $valueAction.properties.outputsLink = @{ uri = 'https://fixture.azure.com/value' } + $downloadedValue = Get-ScimActionEvidence $valueContext $valueAction 'https://management.azure.com/value' $false @() 'Compose' + Assert-Report ($valueContext.failures.Count -eq 0 -and $downloadedValue.contentAvailability.outputs.status -eq 'Collected' -and $downloadedValue.contentAvailability.outputs.httpStatus -eq 200) "Successful HTTP200 $($valueCase.name) download is proven present independently of its value" + Assert-Report ((ConvertTo-ScimCanonicalJson @{ value = $downloadedValue.outputs }) -eq (ConvertTo-ScimCanonicalJson @{ value = $valueCase.value })) "Downloaded $($valueCase.name) retains its exact JSON value" + } + $absentContext = @{ secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); cachedActions = @{} + request = { return $null } } + $absentAction = @{ name = 'Compose_Value'; properties = @{ + status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + } } + $absentEvidence = Get-ScimActionEvidence $absentContext $absentAction 'https://management.azure.com/absent' $false @() 'Compose' + Assert-Report ($absentContext.failures.Count -eq 1 -and $absentEvidence.contentAvailability.outputs.status -eq 'Unavailable') 'A successful Compose with no output property or content link is still a required evidence gap' + $absentAction.properties.outputsLink = @{ uri = 'https://fixture.azure.com/unproven-null' } + $unprovenNull = Get-ScimActionEvidence $absentContext $absentAction 'https://management.azure.com/absent' $false @() 'Compose' + Assert-Report ($unprovenNull.contentAvailability.outputs.status -eq 'Unavailable' -and $absentContext.failures.Count -gt 1) 'An unproven null content response is not silently promoted to a valid JSON null' + $absentContext.request = { param($uri, $arm, $receipt) $receipt.httpStatus = 200; return $null } + $nullArmMetadata = Invoke-ScimEvidenceRequest $absentContext 'https://management.azure.com/null-metadata' + Assert-Report ($null -eq $nullArmMetadata -and $absentContext.failures[-1].reason -match 'without successful HTTP content-response proof') 'Successful HTTP status does not make null ARM metadata a valid workflow or action description' + $absentContext.request = { throw 'Fixture content download failed.' } + $failedDownload = Get-ScimActionEvidence $absentContext $absentAction 'https://management.azure.com/failed-download' $false @() 'Compose' + Assert-Report ($failedDownload.contentAvailability.outputs.status -eq 'Unavailable' -and @($absentContext.failures | Where-Object reason -eq 'Fixture content download failed.').Count -eq 1) 'An actual content download failure remains unavailable and retains its request failure evidence' + $securedNull = Get-ScimActionEvidence $absentContext $absentAction 'https://management.azure.com/secured' $false @('outputs') 'Compose' + Assert-Report ($securedNull.contentAvailability.outputs.status -eq 'SecurityExcluded') 'Protected null or absent payloads remain security exclusions rather than collected values' + $cachedNullContext = @{ secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); reusedPayloadCount = 0 + request = { throw 'Collected null unexpectedly requested a download.' } + cachedActions = @{ 'https://management.azure.com/value' = @{ + status = 'Succeeded'; outputs = $null; contentAvailability = @{ outputs = @{ status = 'Collected' } } + } } + } + $cachedNull = Get-ScimActionEvidence $cachedNullContext $absentAction 'https://management.azure.com/value' $false @() 'Compose' + Assert-Report ($cachedNullContext.failures.Count -eq 0 -and $cachedNullContext.reusedPayloadCount -eq 1 -and $cachedNull.contentAvailability.outputs.status -eq 'ReusedSanitized' -and $null -eq $cachedNull.outputs) 'A previously proven collected JSON null is reusable without refetching or becoming unavailable' + + $listedCalls = [System.Collections.Generic.List[string]]::new() + $skippedRepetitions = @(0..60 | ForEach-Object { @{ + name = [string]$_; properties = @{ + status = 'Skipped'; code = 'ActionSkipped'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:00Z' + error = @{ code = 'BranchNotExecuted'; message = 'fixture-opaque-secret+/=42' } + repetitionIndexes = @(@{ scopeName = 'Until'; itemIndex = $_ }) + } + } }) + $listedContext = @{ + secrets = @($secret); failures = [System.Collections.Generic.List[object]]::new(); cachedActions = @{} + requestMetricsEnabled = $true; requestMetrics = @{}; completedRepetitionCount = 0 + request = { + param($uri, $arm) + $listedCalls.Add($uri) + if ($uri -match '/repetitions\?') { return @{ value = $skippedRepetitions } } + throw 'Unexpected detail or payload request for skipped repetition.' + }.GetNewClosure() + } + $listedAction = @{ name = 'Acquire_Token_Skipped'; properties = @{ status = 'Skipped'; code = 'ActionSkipped' } } + $listedEvidence = Get-ScimActionEvidence $listedContext $listedAction 'https://management.azure.com/action' $true @() 'Workflow' + Assert-Report ($listedEvidence.repetitions.Count -eq 61 -and $listedCalls.Count -eq 1 -and $listedContext.failures.Count -eq 0) 'All 61 skipped repetitions are retained with one listing and no redundant detail or payload requests' + Assert-Report (@($listedEvidence.repetitions | Where-Object metadataSource -eq 'RepetitionListing').Count -eq 61) 'Skipped repetition listing provenance is explicit' + Assert-Report ($listedEvidence.repetitions[0].contentAvailability.outputs.status -eq 'NotExecuted' -and $listedEvidence.repetitions[0].timing.durationMilliseconds -eq 0) 'Skipped repetitions explicitly have no execution payload while preserving supplied timing' + Assert-Report ($listedEvidence.repetitions[0].evidence.properties.error.code -eq 'BranchNotExecuted' -and -not (ConvertTo-ScimCanonicalJson $listedEvidence).Contains($secret)) 'Skipped listing errors and repetition indexes are retained with credential redaction' + Assert-Report ($listedContext.requestMetrics.RepetitionList.count -eq 1 -and -not $listedContext.requestMetrics.Contains('RepetitionDetail')) 'Request diagnostics distinguish listing work from avoided detail requests' + $completeListed = @{ name = '0'; properties = @{ + status = 'Failed'; code = 'BadRequest'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + error = @{ code = 'ObservedFailure'; message = 'Explicit failed probe.' } + inputs = @{ method = 'GET' }; outputsLink = @{ uri = 'https://fixture.azure.com/listed-output' } + } } + $listingCalls = [System.Collections.Generic.List[string]]::new() + $fullListedContext = @{ + secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); cachedActions = @{} + request = { + param($uri, $arm) + $listingCalls.Add($uri) + if ($uri -match '/repetitions\?') { return @{ value = @($completeListed) } } + if ($uri -eq 'https://fixture.azure.com/listed-output') { return @{ statusCode = 400; body = @{ result = 'FAILED' } } } + throw 'Complete listing unexpectedly requested repetition details.' + }.GetNewClosure() + } + $fullListedEvidence = Get-ScimActionEvidence $fullListedContext @{ name = 'Http'; properties = @{ status = 'Failed'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' } } 'https://management.azure.com/http' $true @() 'Http' + Assert-Report ($fullListedEvidence.repetitions.Count -eq 1 -and $listingCalls.Count -eq 2 -and $fullListedContext.failures.Count -eq 0) 'Complete executed listing reuses metadata while still collecting the actual linked payload' + Assert-Report ($fullListedEvidence.repetitions[0].outputs.statusCode -eq 400 -and $fullListedEvidence.repetitions[0].evidence.properties.error.code -eq 'ObservedFailure') 'Reused listing does not truncate failed execution evidence' + $completeListed.properties.Remove('outputsLink') + $listingCalls.Clear() + $fullListedContext.request = { + param($uri, $arm) + $listingCalls.Add($uri) + if ($uri -match '/repetitions\?') { return @{ value = @($completeListed) } } + if ($uri -match '/repetitions/0\?') { + $detail = @{ name = '0'; properties = $completeListed.properties.Clone() } + $detail.properties.outputs = @{ statusCode = 400; body = @{ result = 'FAILED' } } + return $detail + } + throw 'Unexpected incomplete-listing fixture request.' + }.GetNewClosure() + $detailEvidence = Get-ScimActionEvidence $fullListedContext @{ name = 'Http'; properties = @{ status = 'Failed'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' } } 'https://management.azure.com/http' $true @() 'Http' + Assert-Report ($detailEvidence.repetitions[0].metadataSource -eq 'RepetitionDetail' -and $listingCalls.Count -eq 2 -and $detailEvidence.repetitions[0].outputs.statusCode -eq 400) 'Incomplete executed listing still fetches details rather than treating missing evidence as complete' + $pollCalls = [System.Collections.Generic.List[string]]::new() + $pollRepetitions = @(0..60 | ForEach-Object { + $props = @{ + status = if ($_ -eq 60) { 'Failed' } else { 'Succeeded' } + startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + inputsLink = @{ uri = "https://fixture.azure.com/poll/$_/inputs" } + outputsLink = @{ uri = "https://fixture.azure.com/poll/$_/outputs" } + repetitionIndexes = @(@{ scopeName = 'Poll_Until'; itemIndex = $_ }) + retryHistory = @(@{ code = 'RetryObserved'; statusCode = 429 }) + } + if ($_ -eq 60) { + $props.code = 'BadRequest'; $props.error = @{ code = 'NestedPollFailure'; message = 'Failure access_token=raw-poll-error' } + } + @{ name = [string]$_; properties = $props } + }) + $pollContext = @{ + secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); cachedActions = @{} + request = { + param($uri, $arm) + $pollCalls.Add($uri) + if ($uri -match '/repetitions\?') { return @{ value = $pollRepetitions } } + if ($uri -eq 'https://fixture.azure.com/poll/60/inputs') { return @{ method = 'GET'; uri = 'https://scim.example.invalid/Users'; headers = @{ Authorization = 'Bearer raw-poll-auth' } } } + if ($uri -eq 'https://fixture.azure.com/poll/60/outputs') { return @{ statusCode = 400; body = @{ message = 'Failed poll response'; password = 'raw-poll-password' } } } + if ($uri -eq 'https://fixture.azure.com/top-inputs') { return @{ method = 'GET'; uri = 'https://scim.example.invalid/Users' } } + if ($uri -eq 'https://fixture.azure.com/top-outputs') { return @{ statusCode = 200; body = @{ Resources = @() } } } + throw 'Successful poll repetition unexpectedly requested payload or detail.' + }.GetNewClosure() + } + $pollAction = @{ name = 'Poll_Users'; properties = @{ + status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:01:01Z' + inputsLink = @{ uri = 'https://fixture.azure.com/top-inputs' }; outputsLink = @{ uri = 'https://fixture.azure.com/top-outputs' } + } } + $pollEvidence = Get-ScimActionEvidence $pollContext $pollAction 'https://management.azure.com/poll' $true @() 'Http' + Assert-Report ($pollEvidence.repetitions.Count -eq 61 -and $pollContext.failures.Count -eq 0) 'Failure-focused scope retains all sixty successful poll repetitions and the failed repetition without evidence gaps' + Assert-Report ($pollCalls.Count -eq 5 -and @($pollCalls | Where-Object { $_ -match '/poll/\d+/(inputs|outputs)$' }).Count -eq 2) 'Sixty successful repetitions issue zero of their 120 payload GETs; exactly one failed repetition still fetches both payloads, alongside top-level payloads and one listing' + Assert-Report (-not @($pollCalls | Where-Object { $_ -match '/repetitions/\d+\?' }).Count) 'Successful full listing timing/status is sufficient without repetition detail GETs' + $successfulPolls = @($pollEvidence.repetitions | Where-Object status -eq 'Succeeded') + Assert-Report (@($successfulPolls | Where-Object { $_.timing.durationMilliseconds -eq 1000 -and $_.contentAvailability.inputs.status -eq 'NotCollectedByScope' -and $_.contentAvailability.outputs.status -eq 'NotCollectedByScope' }).Count -eq 60) 'Every successful poll retains exact timing and explicitly declares omitted request/response bytes' + Assert-Report ($successfulPolls[0].evidence.properties.retryHistory[0].statusCode -eq 429 -and $successfulPolls[0].evidence.properties.repetitionIndexes.Count -eq 1) 'Successful poll retry errors and loop indexes are not discarded with payloads' + $failedPoll = @($pollEvidence.repetitions | Where-Object status -eq 'Failed')[0] + Assert-Report ($failedPoll.inputs.method -eq 'GET' -and $failedPoll.outputs.statusCode -eq 400 -and $failedPoll.evidence.properties.error.code -eq 'NestedPollFailure') 'Failed poll retains sanitized request, response body, status and nested error evidence' + Assert-Report ($pollEvidence.outputs.statusCode -eq 200 -and $pollEvidence.payloadCollection -eq 'TopLevelDiagnosticPayloads') 'Successful top-level action payloads remain in scope' + Assert-Report (-not (ConvertTo-ScimCanonicalJson $pollEvidence).Contains('raw-poll-')) 'Failure-focused collection preserves credential redaction for failed poll payloads and errors' + Assert-Report (Test-ScimRepetitionPayloadRequired 'Compose' 'Final_TestResults' @{ status = 'Succeeded' }) 'Successful semantic Compose repetitions remain payload-bearing' + Assert-Report (Test-ScimRepetitionPayloadRequired 'Workflow' 'Call_Child' @{ status = 'Succeeded' }) 'Successful child calls retain payloads needed for exact child-run discovery' + Assert-Report (Test-ScimRepetitionPayloadRequired 'Http' 'Probe' @{ status = 'Succeeded'; outputs = @{ body = @{ testResult = 'FAILED' } } }) 'Explicit semantic results remain in scope even when runtime status succeeded' + Assert-Report (Test-ScimRepetitionPayloadRequired 'Http' 'Probe' @{ status = 'TimedOut' }) 'Timed-out repetition payloads are never excluded by the successful-poll scope' + $semanticCacheContext = @{ + secrets = @(); failures = [System.Collections.Generic.List[object]]::new(); reusedPayloadCount = 0 + cachedActions = @{ 'https://management.azure.com/poll/repetitions/known-semantic' = @{ + status = 'Succeeded'; inputs = @{ method = 'GET' }; outputs = @{ body = @{ result = 'FAILED' } } + contentAvailability = @{ inputs = @{ status = 'Collected' }; outputs = @{ status = 'Collected' } } + } } + request = { throw 'Previously collected semantic payload unexpectedly requested network access.' } + } + $semanticCached = Get-ScimActionEvidence $semanticCacheContext @{ name = 'known-semantic'; properties = @{ + status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + } } 'https://management.azure.com/poll/repetitions/known-semantic' $false @() 'Http' -IsRepetition -RepetitionActionName 'Poll_Users' + Assert-Report ($semanticCached.semanticResult -eq 'FAILED' -and $semanticCacheContext.reusedPayloadCount -eq 2 -and $semanticCacheContext.failures.Count -eq 0) 'Resume preserves previously collected semantic-failure payloads even for successful HTTP repetitions whose current listing omits links' + $missingPollContext = $pollContext.Clone() + $missingPollContext.failures = [System.Collections.Generic.List[object]]::new() + [void](Get-ScimActionEvidence $missingPollContext @{ name = '60'; properties = @{ + status = 'TimedOut'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z' + } } 'https://management.azure.com/poll/repetitions/60' $false @() 'Http' -IsRepetition -RepetitionActionName 'Poll_Users') + Assert-Report ($missingPollContext.failures.Count -eq 2) 'Unavailable timed-out repetition request and response are required evidence failures rather than intentional exclusions' + $loopCheckpointPath = Join-Path $fixtureDirectory 'loop-checkpoint.json' + $loopWorkflow = @{ name = 'Fixture_Workflow'; runId = 'fixture-loop-run'; actions = @() } + $loopReport = @{ workflows = @($loopWorkflow); runtime = @{}; metadata = $secret } + $loopContext = $listedContext.Clone() + $loopContext.report = $loopReport; $loopContext.checkpointPath = $loopCheckpointPath + $loopContext.activeWorkflow = $loopWorkflow; $loopContext.activeActionUri = 'https://management.azure.com/action' + $loopContext.completedRepetitionCount = 0; $loopContext.lastCheckpointUtc = [datetimeoffset]::UtcNow.AddSeconds(-31) + $loopInterrupted = $null + try { + function Write-Progress { param($Activity, $Status, [switch] $Completed) throw 'Fixture repetition checkpoint interruption.' } + [void](Get-ScimActionEvidence $loopContext $listedAction 'https://management.azure.com/action' $true @() 'Workflow') + } catch { $loopInterrupted = $_.Exception.Message } + finally { Remove-Item Function:\Write-Progress -ErrorAction SilentlyContinue } + $loopCheckpoint = Get-Content -LiteralPath $loopCheckpointPath -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($loopInterrupted -match 'repetition checkpoint interruption' -and $loopCheckpoint.workflows[0].actions[0].repetitions.Count -eq 1) 'A long action checkpoints its first completed repetition once the interval expires, without waiting for ten slow requests' + Assert-Report ($loopCheckpoint.collection.completedActionCount -eq 0 -and $loopCheckpoint.collection.completedRepetitionCount -eq 1 -and -not $loopCheckpoint.galleryReadiness.ready) 'Partial repetition progress is distinguished from completed actions and never claims readiness' + Assert-Report (-not (Get-Content -LiteralPath $loopCheckpointPath -Raw).Contains($secret)) 'Incremental repetition checkpoint sanitizes metadata and all collected repetition evidence' + $loopContext.completedRepetitionCount = 20 + Save-ScimCollectionCheckpoint $loopContext $loopReport $loopCheckpointPath -RepetitionUpdate + $batchedLoopCheckpoint = Get-Content -LiteralPath $loopCheckpointPath -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($batchedLoopCheckpoint.collection.completedRepetitionCount -eq 1) 'Rapid repetitions do not repeatedly rewrite a growing checkpoint before the thirty-second interval' + Save-ScimCollectionCheckpoint $loopContext $loopReport $loopCheckpointPath -Force + $forcedLoopCheckpoint = Get-Content -LiteralPath $loopCheckpointPath -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($forcedLoopCheckpoint.collection.completedRepetitionCount -eq 20) 'Workflow-boundary forced checkpoint persists progress regardless of the batching interval' + Remove-Item -LiteralPath $loopCheckpointPath + $snapshot = Get-ScimSanitizedReportSnapshot $loopReport @($secret) + Assert-Report ([object]::ReferenceEquals($snapshot.workflows, $loopReport.workflows) -and $snapshot.metadata -eq '[REDACTED]') 'Checkpoint sanitizes new report metadata without rescanning the already-sanitized workflow inventory' + + $actionProperties = @{ status = 'Succeeded'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:02Z' } + $workflowNames = @('Orchestrator_Workflow', 'Initialization_Workflow', 'UserTests_Workflow', 'GroupTests_Workflow', 'SCIMTests_Workflow', 'SCIMToken_Workflow', 'Authentication_Workflow') + foreach ($wfName in $workflowNames) { + $actions = @{} + if ($wfName -eq 'Orchestrator_Workflow') { + foreach ($child in $workflowNames | Where-Object { $_ -notin @('Orchestrator_Workflow', 'SCIMToken_Workflow', 'Authentication_Workflow') }) { + $actions["Call_$child"] = @{ type = 'Workflow'; inputs = @{ host = @{ workflow = @{ id = $child } } } } + } + $actions.Final_TestResults = @{ type = 'Compose'; inputs = @{ testResults = @( + @{ testName = 'Create_User_Test' }, @{ testName = 'OAuth2_Client_Credentials_Test' }, + @{ testName = 'Workload_Identity_Federation_Test' } + ) } } + } elseif ($wfName -eq 'UserTests_Workflow') { + $actions.Create_User_Test = @{ type = 'Scope'; actions = @{} } + $actions.Loop = @{ type = 'Foreach'; actions = @{ Nested_Failure = @{ type = 'Http' } } } + $actions.Loop.actions.Call_Auth = @{ type = 'Workflow'; inputs = @{ host = @{ workflow = @{ id = 'SCIMToken_Workflow' } } } } + $actions.Loop.actions.Nested_Until = @{ type = 'Until'; actions = @{ + Call_Secure_Auth = @{ + type = 'Workflow'; inputs = @{ host = @{ workflow = @{ id = 'SCIMToken_Workflow' } } } + runtimeConfiguration = @{ secureData = @{ properties = @('inputs', 'outputs') } } + } + } } + $actions.Standalone_Until = @{ type = 'Until'; actions = @{ Until_Poll = @{ type = 'Http' } } } + } elseif ($wfName -eq 'SCIMTests_Workflow') { + $actions.Call_Authentication = @{ type = 'Workflow'; inputs = @{ host = @{ workflow = @{ id = 'Authentication_Workflow' } } } } + } elseif ($wfName -eq 'SCIMToken_Workflow') { + $actions.Token_Request = @{ type = 'Http'; runtimeConfiguration = @{ secureData = @{ properties = @('inputs', 'outputs') } } } + } + @{ definition = @{ actions = $actions } } | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath (Join-Path $fixtureDirectory "$wfName.json") + } + $agentPath = Join-Path $fixtureDirectory 'agent.md' + Set-Content -LiteralPath $agentPath -Value 'Offline fixture agent' + $versionPath = Join-Path $fixtureDirectory 'VERSION' + Set-Content -LiteralPath $versionPath -Value '14.0' + # A self-contained source repository proves provenance without depending on parent edits. + & git -C $fixtureDirectory init --quiet + & git -C $fixtureDirectory add --all + & git -C $fixtureDirectory -c user.name=Fixture -c user.email=fixture@example.invalid -c commit.gpgsign=false commit --quiet -m 'Offline reporting fixture' + if ($LASTEXITCODE -ne 0) { throw 'Fixture Git initialization failed.' } + + $state = @{ + core = 'PASSED'; oauth = 'PASSED'; wif = 'PASSED'; overall = 'PASSED'; auth = 'PASSED'; bearer = $null; version = $null; policy = $null; selected = $null; omitWif = $false + deny = $false; omitCore = $false; omitAuth = $false; omitCorrelation = $false + calls = [System.Collections.Generic.List[string]]::new() + } + $request = { + param($uri, $arm) + $state.calls.Add($uri) + if ($state.deny -and $uri -match 'GroupTests_Workflow/runs') { throw '403 Forbidden client_secret=raw-live-denied' } + if ($uri -match '/sites/[^/]+\?') { return @{ identity = @{ type = 'SystemAssigned'; principalId = 'fixture-principal' } } } + if ($uri -match '/SCIMToken_Workflow/runs\?') { + return @{ value = @(@{ name = 'exact-auth-secure-first'; properties = @{ correlation = @{ clientTrackingId = 'fixture-tracking' } } }); nextLink = 'https://management.azure.com/token-run-page2' } + } + if ($uri -eq 'https://management.azure.com/token-run-page2') { + return @{ value = @(@{ name = 'exact-auth-secure-second'; properties = @{ correlation = @{ clientTrackingId = 'fixture-tracking' } } }) } + } + if ($uri -match '/actions/Call_(?:Secure_)?Auth/repetitions/([01])\?') { + $index = $Matches[1] + $props = $actionProperties.Clone() + if ($uri -match '/Call_Secure_Auth/') { + $props.outputs = @{ body = 'raw-secured-repetition-body' } + $props.outputsLink = @{ uri = 'https://fixture.azure.com/forbidden-secure-repetition' } + } else { + $props.outputs = @{ headers = @{ 'x-ms-workflow-run-id' = "exact-auth-$index" } } + } + return @{ name = $index; properties = $props } + } + if ($uri -match '/actions/Call_(?:Secure_)?Auth/repetitions\?') { return @{ value = @(@{ name = '0' }, @{ name = '1' }) } } + if ($uri -match '/(?:scopeRepetitions|repetitions)/0\?') { + return @{ name = '0'; properties = @{ status = 'Failed'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z'; inputs = @{ method = 'GET'; uri = 'https://scim.example.invalid/Users' }; error = @{ code = 'FixtureFailure'; message = 'api_key=raw-repetition-secret' }; outputsLink = @{ uri = 'https://fixture.azure.com/repetition?sig=raw-repetition-sas' } } } + } + if ($uri -match '/(?:scopeRepetitions|repetitions)\?') { return @{ value = @(@{ name = '0' }) } } + if ($uri -match '^https://fixture.azure.com/repetition') { return @{ statusCode = 500; body = @{ message = 'Nested failure'; password = 'raw-body-password' }; headers = @{ Authorization = 'Bearer raw-output-header'; 'x-ms-request-id' = 'repetition-correlation' } } } + if ($uri -match '/actions\?') { + $wf = [regex]::Match($uri, '/workflows/([^/]+)/').Groups[1].Value + $list = [System.Collections.Generic.List[object]]::new() + if ($wf -eq 'Orchestrator_Workflow') { + foreach ($child in $workflowNames | Where-Object { $_ -notin @('Orchestrator_Workflow', 'SCIMToken_Workflow', 'Authentication_Workflow') }) { + $props = $actionProperties.Clone(); $props.outputs = @{ headers = @{ 'x-ms-workflow-run-id' = "exact-$child" } } + $list.Add(@{ name = "Call_$child"; properties = $props }) + } + $props = $actionProperties.Clone() + $props.outputs = @{ + overallResult = $state.overall; authenticationModelResult = $state.auth + testResults = @( + @{ testName = 'Create_User_Test'; testCategory = 'Mandatory'; testResult = $state.core }, + @{ testName = 'OAuth2_Client_Credentials_Test'; testCategory = 'Authentication'; testResult = $state.oauth }, + @{ testName = 'Workload_Identity_Federation_Test'; testCategory = 'Authentication'; testResult = $state.wif } + ) + } + if ($state.bearer) { $props.outputs.BearerConnectivityTestOutputs = @{ overallResult = $state.bearer; result = $state.bearer } } + if ($state.version) { $props.outputs.logicAppVersion = $state.version } + if ($state.policy) { $props.outputs.authenticationPolicy = $state.policy } + if ($state.selected) { $props.outputs.supportedAuthenticationMethods = $state.selected } + if ($state.omitWif) { $props.outputs.testResults = @($props.outputs.testResults | Where-Object testName -ne 'Workload_Identity_Federation_Test') } + if ($state.omitCore) { $props.outputs.testResults = @($props.outputs.testResults | Where-Object testName -ne 'Create_User_Test') } + if ($state.omitAuth) { $props.outputs.Remove('authenticationModelResult') } + $list.Add(@{ name = 'Final_TestResults'; properties = $props }) + } elseif ($wf -eq 'UserTests_Workflow') { + $list.Add(@{ name = 'Create_User_Test'; properties = $actionProperties.Clone() }) + $list.Add(@{ name = 'Loop'; properties = $actionProperties.Clone() }) + $list.Add(@{ name = 'Nested_Failure'; properties = $actionProperties.Clone() }) + $untilProperties = $actionProperties.Clone(); $untilProperties.iterationCount = 1 + $list.Add(@{ name = 'Standalone_Until'; properties = $untilProperties }) + $list.Add(@{ name = 'Until_Poll'; properties = $actionProperties.Clone() }) + $props = $actionProperties.Clone() + $list.Add(@{ name = 'Call_Auth'; properties = $props }) + $props = $actionProperties.Clone(); $props.outputs = @{ body = 'raw-secured-call-body' } + $props.outputsLink = @{ uri = 'https://fixture.azure.com/forbidden-secure-call' } + $list.Add(@{ name = 'Call_Secure_Auth'; properties = $props }) + } elseif ($wf -eq 'SCIMToken_Workflow') { + $props = $actionProperties.Clone() + $props.inputs = @{ body = 'raw-secure-input' }; $props.outputs = @{ body = 'raw-secure-output' } + $props.inputsLink = @{ uri = 'https://fixture.azure.com/forbidden-secure-input' } + $props.outputsLink = @{ uri = 'https://fixture.azure.com/forbidden-secure-output' } + $props.retryHistory = @(@{ statusCode = 429; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:01Z'; body = 'raw-secured-retry-body' }) + $props.error = @{ code = 'FixtureTokenRetry'; message = 'raw-secured-token-error' } + $list.Add(@{ name = 'Token_Request'; properties = $props }) + } elseif ($wf -eq 'SCIMTests_Workflow') { + $props = $actionProperties.Clone(); $props.outputs = @{ headers = @{ 'x-ms-workflow-run-id' = 'exact-authentication' } } + $list.Add(@{ name = 'Call_Authentication'; properties = $props }) + } else { $list.Add(@{ name = 'Compose_Result'; properties = $actionProperties.Clone() }) } + return @{ value = @($list) } + } + if ($uri -match '/runs/[^/]+\?') { + $p = $actionProperties.Clone() + if (-not $state.omitCorrelation) { $p.correlation = @{ clientTrackingId = 'fixture-tracking' } } + if ($uri -match 'exact-auth-1\?') { $p.error = @{ code = 'RetryObserved'; message = 'Token retry evidence access_token=raw-child-error-token' } } + $p.trigger = @{ name = 'manual'; status = 'Succeeded'; startTime = $p.startTime; endTime = $p.endTime } + return @{ name = ([regex]::Match($uri, '/runs/([^?]+)').Groups[1].Value); properties = $p } + } + throw "Unexpected offline request: $uri" + }.GetNewClosure() + $metadata = @{ + supportedAuthenticationMethods = @('OAuth2ClientCredentials', 'WorkloadIdentityFederation') + source = @{ revision = 'caller-source-revision'; repositoryUrl = 'https://example.invalid/repository'; ignored = 'raw-ignored-source' } + deploymentOverlay = @{ name = 'fixture-overlay'; sha256 = ('a' * 64); ignored = 'raw-ignored-overlay'; workflowHashes = @(@{ name = 'SCIMToken_Workflow'; sha256 = ('b' * 64) }) } + harness = @{ name = 'offline fixture'; model = $null; reasoning = $null; context = $null } + } + $reportArgs = @{ + SubscriptionId = 'fixture-subscription'; ResourceGroupName = 'fixture-group'; LogicAppName = 'fixture-app' + OrchestratorRunId = 'exact-orchestrator'; AgentPath = $agentPath; WorkflowDirectory = $fixtureDirectory + Metadata = $metadata; Request = $request; WorkDirectory = $fixtureDirectory + } + $report = New-ScimValidationReport @reportArgs + Assert-Report $report.completeness.complete 'All required fake evidence collected' + Assert-Report ($report.diagnosticScope.name -eq 'FailureFocused' -and $report.completeness.scope.Contains('NotCollectedByScope') -and $report.completeness.scope.Contains('never claims every payload')) 'Report explicitly defines completeness within failure-focused diagnostic scope rather than claiming every successful poll payload' + Assert-Report $report.galleryReadiness.ready 'Every selected gallery authentication method passes' + Assert-Report ($report.provenance.sourceVersion.value -eq '14.0' -and $report.provenance.agent.sourceVersion -eq '14.0') 'Overall and agent source versions come only from VERSION' + Assert-Report ($report.provenance.sourceVersion.sha256 -eq (Get-FileHash $versionPath).Hash.ToLowerInvariant()) 'VERSION exact bytes are hashed for source provenance' + Assert-Report ($null -eq $report.provenance.executedVersion.value -and $null -eq $report.provenance.comparison.deployedVersionMatchesSource) 'Unknown historical deployment is never rebranded with the current source version' + $historicalArgs = $reportArgs.Clone(); $historicalArgs.Metadata = $metadata.Clone() + $historicalArgs.Metadata.historicalRun = @{ version = '11.0'; sourceVersion = @{ value = '13.0'; sha256 = ('c' * 64) }; authenticationPolicy = 'AnySelected'; selectedMethods = @('OAuth2ClientCredentials', 'WorkloadIdentityFederation') } + $historicalReport = New-ScimValidationReport @historicalArgs + Assert-Report ($historicalReport.provenance.executedVersion.value -eq '11.0' -and $historicalReport.provenance.comparison.deployedVersionMatchesSource -eq $false) 'Historical deployed version and source version remain distinct with an explicit mismatch' + Assert-Report ($historicalReport.historicalRun.metadata.authenticationPolicy -eq 'AnySelected') 'Historical OR policy remains intact alongside the current AND assessment' + Assert-Report ($historicalReport.historicalRun.metadata.sourceVersion.value -eq '13.0' -and $historicalReport.historicalRun.metadata.sourceVersion.sha256 -eq ('c' * 64)) 'Original source VERSION and hash remain distinct from deployed version 11 and current reporting source 14' + Assert-Report ($report.agentEvents.status -eq 'Unavailable' -and $report.agentEvents.unavailableReason) 'Historical reports explicitly lack optional agent events' + Assert-Report ($report.agentRun.timingState -eq 'Unrecorded' -and $null -eq $report.agentRun.timing.durationMilliseconds) 'Unrecorded agent window receives no invented timing' + Assert-Report ($report.workflows.Count -eq 10) "Discovers all seven workflow definitions and every repeated child (observed: $($report.workflows.runId -join ', '))" + Assert-Report (@($report.workflows | Where-Object { $_.name -eq 'Authentication_Workflow' -and $_.runId -eq 'exact-authentication' }).Count -eq 1) 'Seventh synchronous authentication workflow is discovered through its nested exact child call' + Assert-Report ($report.orchestratorRunId -eq 'exact-orchestrator') 'Exact orchestrator ID preserved' + Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-0').Count -eq 1) 'First repeated nested child ID preserved' + Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-1').Count -eq 1) 'Second repeated nested child ID preserved' + Assert-Report (@($report.childRunAssociations | Where-Object { $_.parentActionName -eq 'Call_Auth' -and $_.repetitionName -eq '1' }).Count -eq 1) 'Exact repetition association preserved' + Assert-Report (@($report.workflows | Where-Object { $_.runId -eq 'exact-auth-secure-second' -and $_.discovery -match 'Correlation-only' }).Count -eq 1) 'Secured correlation inventory is explicitly not an exact parent mapping' + Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-1')[0].timing.durationMilliseconds -eq 2000) 'Nested child timing collected' + Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-1')[0].run.properties.error.code -eq 'RetryObserved') 'Nested child error retained safely' + Assert-Report ($report.completeness.securityExcludedCount -gt 0 -and $report.securityExclusions.Count -eq 1) 'Secured child evidence is excluded, not falsely incomplete' + Assert-Report ($report.suppliedSource.metadata.revision -eq 'caller-source-revision' -and $report.provenance.sourceGit.revision -ne 'caller-source-revision') 'Caller source claims do not override observed source revision' + Assert-Report ($report.deploymentOverlay.metadata.workflowHashes[0].sha256 -eq ('b' * 64)) 'Allowlisted deployment-overlay hashes retained' + Assert-Report ($report.provenance.agent.sha256 -eq (Get-FileHash $agentPath).Hash.ToLowerInvariant()) 'Agent SHA256' + Assert-Report (-not $report.provenance.sourceGit.dirty) 'Clean source revision recorded' + Assert-Report ($null -eq $report.harness.model.value -and $report.harness.model.unavailableReason) 'Unknown model has explicit reason' + Assert-Report ($report.tests[0].scopeTimings.Count -eq 1) 'Test scope timing matched' + $loop = @($report.workflows | Where-Object name -eq 'UserTests_Workflow')[0].actions | Where-Object name -eq 'Nested_Failure' + Assert-Report ($loop.repetitions[0].status -eq 'Failed') 'Nested failure repetition retained' + Assert-Report (-not @($state.calls | Where-Object { $_ -match '/Standalone_Until/(?:scopeRepetitions|repetitions)' }).Count) 'Until containers do not call runtime endpoints proven unsupported in live probes' + Assert-Report (-not @($state.calls | Where-Object { $_ -match '/Nested_Until/(?:scopeRepetitions|repetitions)' }).Count) 'Nested Until containers also avoid unsupported container repetition endpoints' + Assert-Report (@($state.calls | Where-Object { $_ -match '/Until_Poll/repetitions\?' }).Count -gt 0) 'Until child action repetitions remain fully collected' + Assert-Report (@($state.calls | Where-Object { $_ -match '/Loop/scopeRepetitions\?' }).Count -gt 0) 'Foreach containers use the scopeRepetitions API' + Assert-Report ($loop.repetitions[0].outputs.headers.'x-ms-request-id' -eq 'repetition-correlation') 'Repetition correlation retained' + Assert-Report (-not @($state.calls | Where-Object { $_ -match 'forbidden-secure' }).Count) 'SecureData links never fetched' + $protected = @($report.workflows | Where-Object name -eq 'SCIMToken_Workflow')[0].actions[0] + Assert-Report ($protected.securityExclusions.Count -eq 2) 'SecureData exclusions explicitly recorded' + Assert-Report ($protected.evidence.properties.retryHistory[0].statusCode -eq 429) 'Token-action safe retry status retained' + Assert-Report ($protected.evidence.properties.error.code -eq 'FixtureTokenRetry') 'Token-action safe error code retained without protected error detail' + Assert-Report ($loop.repetitions[0].evidence.properties.error.code -eq 'FixtureFailure') 'Structured error code retained' + $serializedReport = ConvertTo-ScimCanonicalJson $report + Assert-Report (-not $serializedReport.Contains('raw-')) 'No raw token/header/body/error/SAS fixture persisted' + $second = New-ScimValidationReport @reportArgs + Assert-Report ((ConvertTo-ScimCanonicalJson $second) -eq $serializedReport) 'Repeated export is deterministic' + $resumeArgs = $reportArgs.Clone(); $resumeArgs.PreviousReport = $report + $beforePayloadCalls = @($state.calls | Where-Object { $_ -match '^https://fixture.azure.com/repetition' }).Count + $beforeRunCalls = @($state.calls | Where-Object { $_ -match '/runs/[^/]+\?' }).Count + $resumed = New-ScimValidationReport @resumeArgs + Assert-Report ($resumed.resume.reusedPayloadCount -gt 0 -and $resumed.completeness.complete) 'Terminal report resume reuses previously collected sanitized payloads' + Assert-Report (@($state.calls | Where-Object { $_ -match '^https://fixture.azure.com/repetition' }).Count -eq $beforePayloadCalls) 'Resume does not re-download successful immutable payload links' + Assert-Report (@($state.calls | Where-Object { $_ -match '/runs/[^/]+\?' }).Count -gt $beforeRunCalls) 'Resume still reads current run headers rather than replaying stale statuses' + $partialPrevious = ConvertTo-ScimCanonicalJson $report | ConvertFrom-Json -AsHashtable + $partialUser = @($partialPrevious.workflows | Where-Object name -eq 'UserTests_Workflow')[0] + $partialAction = @($partialUser.actions | Where-Object name -eq 'Nested_Failure')[0] + $partialAction.repetitions[0].outputs = $null + $partialAction.repetitions[0].contentAvailability.outputs.status = 'Unavailable' + $partialPrevious.evidenceFailures = @(@{ source = 'Old fixture failure'; reason = 'Old unavailable payload'; required = $true }) + $resumeArgs.PreviousReport = $partialPrevious + $retriedResume = New-ScimValidationReport @resumeArgs + Assert-Report (@($state.calls | Where-Object { $_ -match '^https://fixture.azure.com/repetition' }).Count -gt $beforePayloadCalls -and $retriedResume.completeness.complete) 'Resume retries previously missing payloads and recomputes failures' + Assert-Report (-not @($retriedResume.evidenceFailures | Where-Object source -eq 'Old fixture failure').Count) 'Resume does not inherit obsolete evidence failures' + $wrongResumeArgs = $resumeArgs.Clone(); $wrongResumeArgs.OrchestratorRunId = 'different-root' + $wrongResumeReason = $null + try { [void](New-ScimValidationReport @wrongResumeArgs) } catch { $wrongResumeReason = $_.Exception.Message } + Assert-Report ($wrongResumeReason -match 'does not match') 'Resume refuses a different root or Azure site' + $checkpointDirectory = Join-Path $fixtureDirectory 'interrupted' + [void][IO.Directory]::CreateDirectory($checkpointDirectory) + $checkpointPath = Join-Path $checkpointDirectory 'validation-run-details.checkpoint.json' + $canonicalPath = Join-Path $checkpointDirectory 'validation-run-details.json' + Set-Content -LiteralPath $canonicalPath -Value 'existing canonical report' + $checkpointArgs = $reportArgs.Clone(); $checkpointArgs.CheckpointPath = $checkpointPath + $checkpointFailure = $null + try { + function Write-Progress { + param($Activity, $Status, [switch] $Completed) + if (Test-Path -LiteralPath $checkpointPath) { + $saved = Get-Content -LiteralPath $checkpointPath -Raw | ConvertFrom-Json -AsHashtable + if (@($saved.workflows | ForEach-Object { $_.actions | Where-Object name -eq 'Nested_Failure' }).Count) { throw 'Fixture collection interruption after checkpoint save.' } + } + } + [void](New-ScimValidationReport @checkpointArgs) + } catch { $checkpointFailure = $_.Exception.Message } + finally { Remove-Item Function:\Write-Progress -ErrorAction SilentlyContinue } + Assert-Report ($checkpointFailure -match 'Fixture collection interruption') 'A controlled interruption occurs after successful evidence is persisted' + $savedCheckpoint = Get-Content -LiteralPath $checkpointPath -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($savedCheckpoint.collection.status -eq 'InProgress' -and -not $savedCheckpoint.completeness.complete -and $savedCheckpoint.galleryReadiness.exitCode -eq 2) 'Interrupted checkpoint never claims completeness or readiness' + Assert-Report ($savedCheckpoint.provenance.sourceVersion.value -eq '14.0' -and $savedCheckpoint.orchestratorRunId -eq 'exact-orchestrator') 'Checkpoint carries source VERSION and exact original run identity' + Assert-Report (-not (Get-Content -LiteralPath $checkpointPath -Raw).Contains('raw-')) 'Incremental checkpoint contains only sanitized evidence' + Assert-Report ((Get-Content -LiteralPath $canonicalPath -Raw).Trim() -eq 'existing canonical report') 'Interrupted collection does not overwrite an existing canonical report' + Assert-Report (-not (Test-Path -LiteralPath "$checkpointPath.pending")) 'Checkpoint replacement leaves no pending scratch file after successful save' + $checkpointArgs.PreviousReport = $savedCheckpoint + $checkpointResumed = New-ScimValidationReport @checkpointArgs + Assert-Report ($checkpointResumed.completeness.complete -and $checkpointResumed.resume.reusedPayloadCount -gt 0) 'A partial interrupted checkpoint resumes successful payloads and completes all remaining evidence' + Write-ScimValidationReport $report $fixtureDirectory -Zip + foreach ($file in @('validation-run-details.json', 'validation-report.md', 'validation-report.zip')) { + Assert-Report (Test-Path -LiteralPath (Join-Path $fixtureDirectory $file)) "Output created: $file" + } + Assert-Report (-not (Get-Content (Join-Path $fixtureDirectory 'validation-report.md') -Raw).Contains('raw-')) 'Human report sanitized' + $agentEventsFile = Join-Path $fixtureDirectory 'agent-events.json' + @( + @{ name = 'Setup'; kind = 'setup'; parentName = 'Agent'; startTime = '2026-01-01T01:00:00+01:00'; endTime = '2026-01-01T00:00:03Z'; status = 'Succeeded'; details = @{ message = 'Configured endpoint'; client_secret = 'raw-agent-client-secret'; embedded = '{"access_token":"raw-agent-embedded-token"}'; literal = $secret } }, + @{ name = 'Cleanup'; kind = 'cleanup'; parentName = 'Agent'; startTime = '2026-01-01T00:00:04Z'; endTime = $null; status = 'Running'; details = 'Authorization: Bearer raw-agent-free-token' }, + @{ name = 'HistoricalStep'; kind = 'step'; parentName = $null; startTime = $null; endTime = $null; status = 'Unknown'; details = $null }, + @{ name = 'MissingStart'; kind = 'step'; parentName = $null; startTime = $null; endTime = '2026-01-01T00:00:05Z'; status = 'Succeeded'; details = $null }, + @{ name = 'InvalidTime'; kind = 'step'; parentName = $null; startTime = 'invalid'; endTime = 'invalid'; status = 'Failed'; details = 'client_secret=raw-agent-error-secret' } + ) | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $agentEventsFile + $metadata.agentEventsPath = 'agent-events.json' + $metadata.agentRun = @{ startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:01:00Z'; status = 'Succeeded' } + $eventArgs = $reportArgs.Clone(); $eventArgs.KnownSecrets = @($secret) + $withEvents = New-ScimValidationReport @eventArgs + Assert-Report ($withEvents.completeness.complete -and $withEvents.galleryReadiness.ready) 'Partial optional agent timings do not invalidate historical Azure evidence' + Assert-Report ($withEvents.agentRun.timing.durationMilliseconds -eq 60000 -and $withEvents.fullRun.timing.durationMilliseconds -eq 2000) 'Agent window is separate from Azure Orchestrator window' + Assert-Report ($withEvents.agentEvents.events[0].timing.durationMilliseconds -eq 3000) 'Explicit event elapsed time is calculated in UTC' + Assert-Report ($withEvents.agentEvents.events[0].timing.startUtc -eq '2026-01-01T00:00:00.0000000Z') 'Agent event timestamps normalized to UTC' + Assert-Report ($withEvents.agentEvents.events[1].timingState -eq 'Open' -and $null -eq $withEvents.agentEvents.events[1].timing.endUtc -and $null -eq $withEvents.agentEvents.events[1].timing.durationMilliseconds) 'Open agent event has no invented end or duration' + Assert-Report ($withEvents.agentEvents.events[2].timingState -eq 'Unrecorded' -and $withEvents.agentEvents.events[2].timing.unavailableReason) 'Unrecorded event timing is explicit' + Assert-Report ($withEvents.agentEvents.events[3].timingState -eq 'MissingStart') 'Missing event start is explicit' + Assert-Report ($withEvents.agentEvents.events[4].timingState -eq 'Invalid') 'Invalid event timing is explicit' + Assert-Report ($withEvents.agentEvents.openEventCount -eq 1 -and $withEvents.agentEvents.missingTimingCount -eq 2 -and $withEvents.agentEvents.invalidEventCount -eq 1) 'Partial agent event counters are accurate' + $agentJson = ConvertTo-ScimCanonicalJson $withEvents + Assert-Report (-not $agentJson.Contains('raw-agent-') -and -not $agentJson.Contains($secret)) 'Agent details and exact known literals are sanitized' + Write-ScimValidationReport $withEvents $fixtureDirectory + $agentMarkdown = Get-Content (Join-Path $fixtureDirectory 'validation-report.md') -Raw + Assert-Report ($agentMarkdown.Contains('Agent and setup timing') -and $agentMarkdown.Contains('Cleanup') -and $agentMarkdown.Contains('Open')) 'Human report exposes agent and setup timing with open-event state' + Assert-Report (-not $agentMarkdown.Contains('raw-agent-') -and -not $agentMarkdown.Contains($secret)) 'Human agent event details are sanitized' + $agentContext = @{ failures = [System.Collections.Generic.List[object]]::new(); secrets = @($secret) } + $missingEvents = Get-ScimAgentEvents $agentContext 'missing-agent-events.json' $fixtureDirectory + Assert-Report ($missingEvents.status -eq 'Unavailable' -and $agentContext.failures.Count -eq 1 -and -not $agentContext.failures[0].required) 'Missing optional event file is explicitly unavailable and not required' + $invalidEventsPath = Join-Path $fixtureDirectory 'invalid-agent-events.json' + Set-Content -LiteralPath $invalidEventsPath -Value '{"not":"an array"}' + $invalidEvents = Get-ScimAgentEvents $agentContext $invalidEventsPath $fixtureDirectory + Assert-Report ($invalidEvents.status -eq 'Unavailable' -and $invalidEvents.unavailableReason -match 'JSON array') 'Agent event source requires a JSON array' + Set-Content -LiteralPath $invalidEventsPath -Value '[]' + Assert-Report ((Get-ScimAgentEvents $agentContext $invalidEventsPath $fixtureDirectory).status -eq 'Unavailable') 'Empty event array does not pretend recorded coverage' + $openWindow = Get-ScimRecordedTiming @{ startTime = '2026-01-01T00:00:00Z'; status = 'Running' } + Assert-Report ($openWindow.timingState -eq 'Open' -and $null -eq $openWindow.timing.durationMilliseconds) 'Open agent run window has no fabricated cleanup time' + $ambiguousWindow = Get-ScimRecordedTiming @{ startTime = '2026-01-01T00:00:00'; endTime = '2026-01-01T00:01:00' } + Assert-Report ($ambiguousWindow.timingState -eq 'Invalid' -and $null -eq $ambiguousWindow.timing.durationMilliseconds) 'Missing time zone is not guessed from the export machine' + Set-Content -LiteralPath $invalidEventsPath -Value '[ raw-agent-malformed-payload' + $malformedEvents = Get-ScimAgentEvents $agentContext $invalidEventsPath $fixtureDirectory + Assert-Report ($malformedEvents.status -eq 'Unavailable' -and -not $malformedEvents.unavailableReason.Contains('raw-agent-malformed-payload')) 'Malformed agent JSON is diagnosed without echoing raw payload snippets' + $metadata.Remove('agentRun'); $metadata.Remove('agentEventsPath') + + $state.wif = 'FAILED' + $andFailure = New-ScimValidationReport @reportArgs + Assert-Report (-not $andFailure.galleryReadiness.ready -and $andFailure.galleryReadiness.exitCode -eq 1) 'OAuth success cannot waive a selected failed federation method' + Assert-Report ($andFailure.authentication.aggregatePolicyMismatch -and $andFailure.authentication.recordedAggregateResult -eq 'PASSED') 'Stale historical aggregate success remains visible but cannot override selected-method AND validation' + Assert-Report ($andFailure.authentication.policy.Contains('static-token connectivity') -and -not $andFailure.authentication.policy.Contains('[REDACTED')) 'Authentication policy explanation stays readable without weakening credential redaction' + Assert-Report ($andFailure.galleryReadiness.reason.Contains('Static-token connectivity alone') -and -not $andFailure.galleryReadiness.reason.Contains('[REDACTED')) 'Static-token-only readiness explanation is retained by the unchanged redactor' + $metadata.applicability = @{ Workload_Identity_Federation_Test = @{ applicable = $false; reason = 'Attempted selected-method waiver' } } + $state.wif = 'NOT_APPLICABLE - credentials missing' + $selectedMissing = New-ScimValidationReport @reportArgs + Assert-Report ($selectedMissing.galleryReadiness.exitCode -eq 2 -and @($selectedMissing.evidenceFailures | Where-Object source -eq 'Selected authentication method WorkloadIdentityFederation').Count -eq 1) 'Selected method with missing credentials or NA is a blocker despite a caller applicability waiver' + $state.omitWif = $true + $selectedAbsent = New-ScimValidationReport @reportArgs + Assert-Report ($selectedAbsent.galleryReadiness.exitCode -eq 2 -and @($selectedAbsent.authentication.methodResults | Where-Object method -eq 'WorkloadIdentityFederation')[0].resultCount -eq 0) 'A selected method with no result is an explicit missing-evidence blocker' + $state.omitWif = $false + $metadata.Remove('applicability') + $metadata.supportedAuthenticationMethods = @('OAuth2ClientCredentials') + $state.wif = 'FAILED' + $oauthOnly = New-ScimValidationReport @reportArgs + Assert-Report ($oauthOnly.galleryReadiness.ready -and @($oauthOnly.tests | Where-Object authenticationMethod -eq 'WorkloadIdentityFederation')[0].classification -eq 'NotApplicable') 'Undeclared failed federation is not applicable while its original recorded result remains intact' + Assert-Report (@($oauthOnly.tests | Where-Object authenticationMethod -eq 'WorkloadIdentityFederation')[0].semanticResult -eq 'FAILED') 'Historical semantic evidence is not rewritten when applicability changes' + $state.selected = @('OAuth2ClientCredentials', 'WorkloadIdentityFederation') + $selectionMismatch = New-ScimValidationReport @reportArgs + Assert-Report ($selectionMismatch.authentication.selectionMismatch -and $selectionMismatch.galleryReadiness.exitCode -eq 2) 'Caller metadata cannot silently drop a selected failing method from recorded execution' + $state.selected = $null + $metadata.supportedAuthenticationMethods = @('Bearer', 'OAuth2ClientCredentials') + $mixedMissing = New-ScimValidationReport @reportArgs + Assert-Report ($mixedMissing.galleryReadiness.exitCode -eq 2) 'Selected static bearer with no connectivity probe evidence blocks a mixed-method report' + $state.bearer = 'FAILED' + Assert-Report ((New-ScimValidationReport @reportArgs).galleryReadiness.exitCode -eq 1) 'Failed bearer connectivity blocks otherwise passing selected OAuth' + $state.bearer = 'PASSED' + $mixedPassing = New-ScimValidationReport @reportArgs + Assert-Report ($mixedPassing.galleryReadiness.ready -and $mixedPassing.authentication.methodResults.Count -eq 2) 'Mixed bearer and OAuth requires both the actual connectivity probe and gallery authentication test to pass' + Assert-Report ($mixedPassing.tests.Count -eq 3 -and $mixedPassing.authentication.bearerConnectivity.overallResult -eq 'PASSED') 'Separate bearer probe does not change the historical named-test inventory' + $metadata.supportedAuthenticationMethods = @('Bearer') + Assert-Report (-not (New-ScimValidationReport @reportArgs).galleryReadiness.ready) 'A passing static bearer connectivity probe alone is still not gallery-ready' + $state.bearer = $null; $state.wif = 'PASSED' + $metadata.supportedAuthenticationMethods = @('OAuth2ClientCredentials', 'WorkloadIdentityFederation') + $state.version = '14.0'; $state.policy = 'AllSelected' + $currentRun = New-ScimValidationReport @reportArgs + Assert-Report ($currentRun.provenance.executedVersion.value -eq '14.0' -and $currentRun.authentication.recordedPolicy -eq 'AllSelected' -and $currentRun.provenance.comparison.deployedVersionMatchesSource) 'New deployed outputs explicitly establish their own version and AllSelected policy' + $state.version = $null; $state.policy = $null + Remove-Item -LiteralPath $versionPath + Assert-Report ((New-ScimValidationReport @reportArgs).galleryReadiness.exitCode -eq 2) 'Missing VERSION provenance blocks readiness instead of inventing a revision' + Set-Content -LiteralPath $versionPath -Value '14.0' + $state.core = 'FAILED raw-scope-semantic' + $failedArgs = $reportArgs.Clone(); $failedArgs.KnownSecrets = @('raw-scope-semantic') + $failed = New-ScimValidationReport @failedArgs + Assert-Report (-not $failed.galleryReadiness.ready -and $failed.galleryReadiness.exitCode -eq 1) 'Succeeded runtime does not override failed semantic core test' + Assert-Report (-not (ConvertTo-ScimCanonicalJson $failed).Contains('raw-scope-semantic')) 'Late scope semantic updates are sanitized before entering the reusable workflow inventory' + $legacyFixture = ConvertTo-ScimCanonicalJson $failed | ConvertFrom-Json -AsHashtable + $legacyEntry = @{ + name = '0'; type = 'Compose'; status = 'Succeeded'; collectionState = 'Completed' + payloadCollection = 'RequiredDiagnosticRepetition'; inputs = ''; outputs = $null; securityExclusions = @() + contentAvailability = @{ inputs = @{ status = 'Collected' }; outputs = @{ status = 'Unavailable' } } + evidence = @{ properties = @{ outputsLink = @{ uri = 'https://fixture.azure.com/legacy-empty'; contentSize = 2 } } } + } + $legacyParent = @{ name = 'Compose_Legacy'; type = 'Compose'; status = 'Succeeded'; collectionState = 'Completed' + repetitionCollection = @{ kind = 'repetitions' }; repetitions = @($legacyEntry) } + @($legacyFixture.workflows | Where-Object name -eq 'Orchestrator_Workflow')[0].actions += $legacyParent + $legacySource = 'https://management.azure.com/subscriptions/fixture-subscription/resourceGroups/fixture-group/providers/Microsoft.Web/sites/fixture-app/hostruntime/runtime/webhooks/workflow/api/management/workflows/Orchestrator_Workflow/runs/exact-orchestrator/actions/Compose_Legacy/repetitions/0/outputs' + $legacyFixture.evidenceFailures += @{ source = $legacySource; reason = 'Successful data-producing action supplied no readable output evidence.'; required = $true } + $legacyFixture.runtime.evidenceRequests = @{ ContentPayload = @{ count = 1; failed = 0 } } + $legacyFixture.completeness.complete = $false; $legacyFixture.completeness.requiredUnavailableCount = 1 + $legacyFixture.galleryReadiness.status = 'Incomplete'; $legacyFixture.galleryReadiness.exitCode = 2 + $legacyPath = Join-Path $fixtureDirectory 'legacy-empty-report.json' + [IO.File]::WriteAllText($legacyPath, (ConvertTo-ScimCanonicalJson $legacyFixture), [Text.UTF8Encoding]::new($false)) + $legacyHash = (Get-FileHash -LiteralPath $legacyPath).Hash.ToLowerInvariant() + $offline = Complete-ScimValidationReport -ReportPath $legacyPath + Assert-Report ($offline.offlineFinalization.correctedOutputCount -eq 1 -and $offline.offlineFinalization.networkRequests -eq 0 -and $offline.completeness.complete) 'Offline finalization repairs the proven empty-string classification without recollection' + Assert-Report ($offline.galleryReadiness.status -eq 'NotReady' -and $offline.galleryReadiness.exitCode -eq 1 -and -not $offline.galleryReadiness.ready) 'Correcting collection completeness never waives recorded functional failures or establishes readiness' + $offlineEntry = @(@($offline.workflows | Where-Object name -eq 'Orchestrator_Workflow')[0].actions | Where-Object name -eq 'Compose_Legacy')[0].repetitions[0] + Assert-Report ($offlineEntry.outputs -is [string] -and $offlineEntry.outputs.Length -eq 0 -and $offlineEntry.contentAvailability.outputs.presenceSource -eq 'OfflineComposeIdentityCorrection') 'Offline proof restores an empty string, not an invented JSON null or success result' + Assert-Report ((Get-FileHash -LiteralPath $legacyPath).Hash.ToLowerInvariant() -eq $legacyHash -and $offline.offlineFinalization.sourceReportSha256 -eq $legacyHash) 'Offline finalization reads the source report immutably and records its exact SHA256' + Assert-Report ((ConvertTo-ScimCanonicalJson $offline.provenance) -eq (ConvertTo-ScimCanonicalJson $legacyFixture.provenance) -and + (ConvertTo-ScimCanonicalJson $offline.tests) -eq (ConvertTo-ScimCanonicalJson $legacyFixture.tests)) 'Offline finalization preserves historical/source provenance and every original semantic test result' + Assert-Report ($offline.offlineFinalization.corrector.fileName -eq 'Export-ScimValidationReport.ps1' -and + $offline.offlineFinalization.corrector.sha256 -eq (Get-FileHash -LiteralPath (Join-Path $PSScriptRoot 'Export-ScimValidationReport.ps1')).Hash.ToLowerInvariant()) 'Offline finalization records the correction implementation hash separately from original run provenance' + $offlineOutputDirectory = Join-Path $fixtureDirectory 'offline-finalized' + Write-ScimValidationReport $offline $offlineOutputDirectory -Zip + $persistedOffline = Get-Content -LiteralPath (Join-Path $offlineOutputDirectory 'validation-run-details.json') -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($persistedOffline.offlineFinalization.sourceReportSha256 -eq $legacyHash -and + $persistedOffline.galleryReadiness.exitCode -eq 1 -and + (Get-Content -LiteralPath (Join-Path $offlineOutputDirectory 'validation-report.md') -Raw).Contains($legacyHash) -and + (Test-Path -LiteralPath (Join-Path $offlineOutputDirectory 'validation-report.zip'))) 'The standard writer persists corrected JSON, Markdown and ZIP into a separate directory with original-report proof' + $legacyFixture.evidenceFailures += @{ source = 'StillMissingComposeOutput'; reason = 'Successful data-producing action supplied no readable output evidence.'; required = $true } + [IO.File]::WriteAllText($legacyPath, (ConvertTo-ScimCanonicalJson $legacyFixture), [Text.UTF8Encoding]::new($false)) + $partiallyOffline = Complete-ScimValidationReport -ReportPath $legacyPath + Assert-Report ($partiallyOffline.completeness.requiredUnavailableCount -eq 1 -and $partiallyOffline.galleryReadiness.exitCode -eq 2) 'Offline correction leaves an unproven missing Compose output incomplete instead of waiving all Compose gaps' + $legacyFixture.runtime.evidenceRequests.ContentPayload.failed = 1 + [IO.File]::WriteAllText($legacyPath, (ConvertTo-ScimCanonicalJson $legacyFixture), [Text.UTF8Encoding]::new($false)) + $offlineReason = $null + try { [void](Complete-ScimValidationReport -ReportPath $legacyPath) } catch { $offlineReason = $_.Exception.Message } + Assert-Report ($offlineReason -match 'zero failed downloads') 'Offline correction refuses to infer valid outputs when content downloads failed' + $legacyFixture.runtime.evidenceRequests.ContentPayload.failed = 0; $legacyEntry.inputs = $null + [IO.File]::WriteAllText($legacyPath, (ConvertTo-ScimCanonicalJson $legacyFixture), [Text.UTF8Encoding]::new($false)) + $offlineReason = $null + try { [void](Complete-ScimValidationReport -ReportPath $legacyPath) } catch { $offlineReason = $_.Exception.Message } + Assert-Report ($offlineReason -match 'sufficient recorded proof') 'Null Compose inputs or output length alone cannot be relabeled as collected empty strings' + $legacyEntry.inputs = ''; $legacyEntry.securityExclusions = @(@{ source = 'outputs'; reason = 'Protected fixture.' }) + [IO.File]::WriteAllText($legacyPath, (ConvertTo-ScimCanonicalJson $legacyFixture), [Text.UTF8Encoding]::new($false)) + $offlineReason = $null + try { [void](Complete-ScimValidationReport -ReportPath $legacyPath) } catch { $offlineReason = $_.Exception.Message } + Assert-Report ($offlineReason -match 'sufficient recorded proof') 'Offline finalization cannot turn protected output bytes into collected evidence' + $state.core = 'PASSED'; $state.oauth = 'SKIPPED'; $state.wif = 'SKIPPED' + Assert-Report (-not (New-ScimValidationReport @reportArgs).galleryReadiness.ready) 'Selected auth skipped is not success' + $state.oauth = 'PASSED'; $metadata.supportedAuthenticationMethods = @('Bearer') + Assert-Report (-not (New-ScimValidationReport @reportArgs).galleryReadiness.ready) 'Bearer-only is not gallery-ready' + $metadata.supportedAuthenticationMethods = @('OAuth2ClientCredentials') + $state.deny = $true + $incomplete = New-ScimValidationReport @reportArgs + Assert-Report (-not $incomplete.completeness.complete -and $incomplete.galleryReadiness.exitCode -eq 2) 'Required evidence denied is incomplete and nonzero' + Assert-Report (-not (ConvertTo-ScimCanonicalJson $incomplete).Contains('raw-live-denied')) 'ARM denial secrets never persisted' + $state.deny = $false + $state.omitCore = $true + $missingCore = New-ScimValidationReport @reportArgs + Assert-Report (-not $missingCore.completeness.complete -and -not $missingCore.galleryReadiness.ready) 'Missing expected core result is incomplete, not passing' + $state.omitCore = $false; $state.omitAuth = $true + Assert-Report ((New-ScimValidationReport @reportArgs).galleryReadiness.exitCode -eq 2) 'Missing authentication model summary is incomplete' + $state.omitAuth = $false + $unavailableArgs = $reportArgs.Clone() + $unavailableArgs.Metadata = $metadata.Clone() + $unavailableArgs.Metadata.preflight = @{ status = 'Passed'; startTime = '2026-01-01T00:00:00Z'; endTime = '2026-01-01T00:00:02Z' } + $unavailableArgs.Request = { param($uri, $arm) throw 'ARM evidence could not be acquired.' } + $unavailableRun = New-ScimValidationReport @unavailableArgs + Assert-Report ($unavailableRun.fullRun.status -eq 'Unavailable' -and $unavailableRun.orchestratorRunId -eq 'exact-orchestrator') 'Requested execution with failed collection is Unavailable, never NotExecuted' + Assert-Report ($null -eq $unavailableRun.fullRun.timing.startUtc -and $null -eq $unavailableRun.fullRun.timing.durationMilliseconds) 'Unavailable Azure run does not borrow passed-preflight timings' + Assert-Report ($unavailableRun.fullRun.unavailableReason -and $unavailableRun.galleryReadiness.exitCode -eq 2) 'Unavailable requested execution has explicit reason and incomplete exit status' + $state.omitCorrelation = $true + $excluded = New-ScimValidationReport @reportArgs + Assert-Report ($excluded.completeness.complete -and $excluded.completeness.securityExcludedCount -gt 0) 'Intentionally hidden child mappings without correlation do not falsely fail completeness' + $metadata.workflowRuns = @( + @{ workflowName = 'SCIMToken_Workflow'; runId = 'exact-auth-secure-first'; parentWorkflowName = 'UserTests_Workflow'; parentRunId = 'exact-UserTests_Workflow'; parentActionName = 'Call_Secure_Auth'; repetitionName = '0' }, + @{ workflowName = 'SCIMToken_Workflow'; runId = 'exact-auth-secure-second'; parentWorkflowName = 'UserTests_Workflow'; parentRunId = 'exact-UserTests_Workflow'; parentActionName = 'Call_Secure_Auth'; repetitionName = '1' } + ) + $suppliedRuns = New-ScimValidationReport @reportArgs + Assert-Report ($suppliedRuns.workflows.Count -eq 10 -and $suppliedRuns.completeness.complete) 'Caller-supplied exact protected child IDs collect all runs without correlation' + Assert-Report (@($suppliedRuns.childRunAssociations | Where-Object attribution -match 'Caller-supplied').Count -eq 2) 'Caller-supplied associations remain distinctly attributed' + $metadata.Remove('workflowRuns'); $state.omitCorrelation = $false + $metadata.supportedAuthenticationMethods = @('WorkloadIdentityFederation') + $metadata.scimAuthenticationMethod = 'WorkloadIdentityFederation' + $state.wif = 'PASSED'; $state.oauth = 'NOT_APPLICABLE' + $wifTransport = New-ScimValidationReport @reportArgs + Assert-Report ($wifTransport.galleryReadiness.ready -and $wifTransport.authentication.scimRequestTransport.value -eq 'WorkloadIdentityFederation') 'WIF-only gallery alternative and direct transport require no OAuth credential evidence' + $blockedArgs = $reportArgs.Clone(); $blockedArgs.OrchestratorRunId = '' + $blocked = New-ScimValidationReport @blockedArgs + Assert-Report ($null -eq $blocked.orchestratorRunId -and $blocked.fullRun.status -eq 'Blocked') 'Blocked export has no fake execution ID' + Assert-Report ($blocked.galleryReadiness.exitCode -eq 2) 'Blocked export returns incomplete' + $na = @(Get-ScimSemanticTests @{ testResults = @(@{ testName = 'Undeclared'; testResult = 'NOT_APPLICABLE'; applicable = $false; reason = 'Not declared' }) } @{} @()) + Assert-Report (-not $na[0].applicable -and $na[0].applicabilityReason -eq 'Not declared') 'Explicit non-applicability preserved' + $noReason = @(Get-ScimSemanticTests @{ testResults = @(@{ testName = 'Undeclared'; testResult = 'NOT_APPLICABLE'; applicable = $false }) } @{} @()) + Assert-Report $noReason[0].applicable 'Non-applicable without reason cannot silently waive a test' + $blockedMetadataPath = Join-Path $fixtureDirectory 'report-metadata.json' + @{ supportedAuthenticationMethods = @('OAuth2ClientCredentials'); preflight = @{ status = 'Blocked'; reason = 'Fixture preflight was blocked.' } } | + ConvertTo-Json -Depth 4 | Set-Content -LiteralPath $blockedMetadataPath + $cliOutput = & (Join-Path $PSHOME 'pwsh.exe') -NoProfile -File (Join-Path $PSScriptRoot 'Export-ScimValidationReport.ps1') ` + -WorkDirectory $fixtureDirectory -AgentPath $agentPath -WorkflowDirectory $fixtureDirectory -MetadataPath $blockedMetadataPath 2>&1 + Assert-Report ($LASTEXITCODE -eq 2) 'CLI blocked execution exits 2 without Azure CLI' + $blockedFile = Get-Content (Join-Path $fixtureDirectory 'validation-run-details.json') -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($null -eq $blockedFile.orchestratorRunId -and $blockedFile.fullRun.status -eq 'Blocked') 'CLI persists honest blocked report' + $launcherPath = Join-Path $fixtureDirectory 'offline-executed-export.ps1' + @' +param($Exporter, $WorkDirectory, $AgentPath, $WorkflowDirectory, $MetadataPath) +$ErrorActionPreference = 'Stop' +$global:failFirstArmRequest = $true +function az { + $global:LASTEXITCODE = 0 + if ($args[0] -eq 'version') { return 'fixture-cli' } + if ($args[1] -eq 'show') { return '{"tenantId":"fixture-tenant","operatorIdentity":{"type":"user","name":"fixture"}}' } + return '{"accessToken":"fixture-standalone-token","tenant":"fixture-tenant","expires_on":4102444800}' +} +function Invoke-RestMethod { + [CmdletBinding()] + param($Uri, $Headers, $Method, $MaximumRedirection, $TimeoutSec, $StatusCodeVariable) + if ($StatusCodeVariable) { Set-Variable -Name $StatusCodeVariable -Value 200 -Scope 1 } + if ($global:failFirstArmRequest) { + $global:failFirstArmRequest = $false + $errorObject = [System.Net.Http.HttpRequestException]::new('Fixture standalone HTTP401') + $errorObject.Data['StatusCode'] = 401 + throw $errorObject + } + $p = @{status='Succeeded';startTime='2026-01-01T00:00:00Z';endTime='2026-01-01T00:00:01Z'} + if ($Uri -match '/sites/[^/]+\?') { return @{identity=@{type='SystemAssigned';principalId='fixture'}} } + if ($Uri -match '/actions\?') { + if ($Uri -match '/Orchestrator_Workflow/') { + $items = @() + foreach ($child in @('Initialization_Workflow','UserTests_Workflow','GroupTests_Workflow','SCIMTests_Workflow')) { + $cp=$p.Clone();$cp.outputs=@{headers=@{'x-ms-workflow-run-id'="cli-$child"}} + $items+=@{name="Call_$child";properties=$cp} + } + $fp=$p.Clone() + $fp.outputs=@{overallResult='PASSED';authenticationModelResult='PASSED';testResults=@( + @{testName='Create_User_Test';testCategory='Mandatory';testResult='PASSED'}, + @{testName='OAuth2_Client_Credentials_Test';testCategory='Authentication';testResult='PASSED'}, + @{testName='Workload_Identity_Federation_Test';testCategory='Authentication';testResult='PASSED'} + )} + $items+=@{name='Final_TestResults';properties=$fp} + return @{value=$items} + } + $p.outputs=@{result='PASSED'} + return @{value=@(@{name='Fixture_Compose';properties=$p})} + } + if ($Uri -match '/runs/[^/]+\?') { + $p.trigger=@{name='manual';status='Succeeded';startTime=$p.startTime;endTime=$p.endTime} + return @{properties=$p} + } + throw 'Unexpected standalone fixture request.' +} +& $Exporter -WorkDirectory $WorkDirectory -AgentPath $AgentPath -WorkflowDirectory $WorkflowDirectory -MetadataPath $MetadataPath ` + -SubscriptionId fixture-subscription -ResourceGroupName fixture-group -LogicAppName fixture-app -TenantId fixture-tenant -OrchestratorRunId cli-root +exit $LASTEXITCODE +'@ | Set-Content -LiteralPath $launcherPath + $executedOutputDirectory = Join-Path $fixtureDirectory 'standalone-executed' + $executedCliOutput = & (Join-Path $PSHOME 'pwsh.exe') -NoProfile -File $launcherPath ` + -Exporter (Join-Path $PSScriptRoot 'Export-ScimValidationReport.ps1') -WorkDirectory $executedOutputDirectory ` + -AgentPath $agentPath -WorkflowDirectory $fixtureDirectory -MetadataPath $blockedMetadataPath 2>&1 + Assert-Report ($LASTEXITCODE -eq 0) 'Standalone executed CLI resolves its private refreshing transport and completes using only offline fixtures' + $standaloneReport = Get-Content (Join-Path $executedOutputDirectory 'validation-run-details.json') -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($standaloneReport.runtime.arm401RetryCount -eq 1 -and $standaloneReport.completeness.complete) "Standalone CLI refreshes a simulated real HTTP401 and records no obsolete failure (retries=$($standaloneReport.runtime.arm401RetryCount), complete=$($standaloneReport.completeness.complete))" + if ($EvidenceDirectory) { + [void][IO.Directory]::CreateDirectory([IO.Path]::GetFullPath($EvidenceDirectory)) + $contract = @{ + purpose = 'Offline report-contract fixture only; not a live validation or gallery-readiness claim' + assertionsPassed = $script:checks + actualSourceVersion = @{ + value = (Get-Content -LiteralPath (Join-Path $PSScriptRoot 'VERSION') -Raw).Trim() + sha256 = (Get-FileHash -LiteralPath (Join-Path $PSScriptRoot 'VERSION') -Algorithm SHA256).Hash.ToLowerInvariant() + } + historicalFixture = @{ + orchestratorRunId = $historicalReport.orchestratorRunId + reportingSourceVersion = $historicalReport.provenance.sourceVersion + observedRunVersion = $historicalReport.provenance.executedVersion + historicalClaims = $historicalReport.historicalRun + deployedVersionMatchesSource = $historicalReport.provenance.comparison.deployedVersionMatchesSource + } + currentFixture = @{ + orchestratorRunId = $currentRun.orchestratorRunId + reportingSourceVersion = $currentRun.provenance.sourceVersion + observedRunVersion = $currentRun.provenance.executedVersion + recordedPolicy = $currentRun.authentication.recordedPolicy + deployedVersionMatchesSource = $currentRun.provenance.comparison.deployedVersionMatchesSource + } + andPolicyFixture = @{ + recordedAggregate = $andFailure.authentication.recordedAggregateResult + policyMismatch = $andFailure.authentication.aggregatePolicyMismatch + methodResults = $andFailure.authentication.methodResults + galleryReadiness = $andFailure.galleryReadiness + } + mixedBearerFixture = @{ + methodResults = $mixedPassing.authentication.methodResults + bearerConnectivity = $mixedPassing.authentication.bearerConnectivity + } + interruptionFixture = @{ + savedStatus = $savedCheckpoint.collection.status + savedReady = $savedCheckpoint.galleryReadiness.ready + resumedPayloadCount = $checkpointResumed.resume.reusedPayloadCount + resumedComplete = $checkpointResumed.completeness.complete + } + } + $contractPath = Join-Path $EvidenceDirectory 'report-contract.json' + [IO.File]::WriteAllText($contractPath, (ConvertTo-ScimCanonicalJson $contract) + "`n", [Text.UTF8Encoding]::new($false)) + $savedContract = Get-Content -LiteralPath $contractPath -Raw | ConvertFrom-Json -AsHashtable + Assert-Report ($savedContract.actualSourceVersion.sha256 -eq (Get-FileHash -LiteralPath (Join-Path $PSScriptRoot 'VERSION')).Hash.ToLowerInvariant()) 'Optional contract artifact carries the actual source VERSION hash, separate from fixture source bytes' + Assert-Report ($savedContract.historicalFixture.observedRunVersion.value -eq '11.0' -and $savedContract.currentFixture.observedRunVersion.value -eq '14.0' -and -not $savedContract.andPolicyFixture.galleryReadiness.ready) 'Persisted offline contract proves historical version separation and selected-method AND failure' + $contract.assertionsPassed = $script:checks + [IO.File]::WriteAllText($contractPath, (ConvertTo-ScimCanonicalJson $contract) + "`n", [Text.UTF8Encoding]::new($false)) + } + Write-Host "PASS: $script:checks offline reporting assertions." +} finally { + if (Test-Path -LiteralPath $fixtureDirectory) { Remove-Item -LiteralPath $fixtureDirectory -Recurse -Force } +} diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/VERSION b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/VERSION index ed9d00f9..fe6b09a7 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/VERSION +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/VERSION @@ -1 +1 @@ -14.0 +15.0 diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md new file mode 100644 index 00000000..50924955 --- /dev/null +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md @@ -0,0 +1,118 @@ +# Portable SCIM validation reporting + +## Executive summary + +This reporting change is dependent on the authentication-first change +documented in [Authentication-Changes.md](Authentication-Changes.md). +Authentication must be reviewed and merged first. This increment does not +change authentication policy, token transport, workflow definitions, or +functional test semantics. + +[VERSION](VERSION) advances from the authentication baseline 14.0 to 15.0, +and `LogicAppVersion` is synchronized. Historical reports keep their original +executed/source versions and hashes; a source version bump does not certify +or relabel an earlier deployment. + +## Scope + +| Surface | Reporting enhancement | +|---|---| +| Canonical agent | Create a unique run folder before questions/cloud operations; record actual lifecycle events. | +| Exporter | Collect exact-run metadata, semantic results, timing, failures, and supported nested histories. | +| Portable output | Sanitized JSON, Markdown, and optional ZIP in separate exact-run/versioned folders. | +| Provenance | Distinguish source, executed version, deployed overlay, hashes, and caller claims. | +| Completeness | Expose required/optional gaps, protected payloads, and intentionally uncollected successful polls. | +| Recovery | Checkpoint sanitized evidence, resume only the same run, and finalize metadata offline when appropriate. | +| Tests | Dependency-free offline reporter tests and agent startup-folder regression coverage. | + +All seven runtime workflow JSON files remain byte-identical to the +authentication baseline. The only parameter-template change is the bundle +version. Authentication is not implemented again in this PR. + +## Evidence flow + +```mermaid +flowchart LR + Run[Exact Orchestrator and associated child runs] --> Collect[Paginated history collection] + Source[VERSION and source/deployment hashes] --> Collect + Events[Recorded lifecycle/environment metadata] --> Collect + Collect --> Redact[Recursive sanitization] + Redact --> JSON[Detailed JSON] + JSON --> Markdown[Readable summary] + JSON --> ZIP[Portable archive] + Markdown --> ZIP + Collect --> Gaps[Explicit gaps and protected exclusions] + Gaps --> JSON +``` + +The exporter must never select the newest run as a substitute for exact +association, infer hidden model settings, invent timestamps, disable token +protection, or turn missing required evidence into success. + +## Data transformations and outcomes + +| Input | Transformation / boundary | +|---|---| +| Azure timestamps | Normalize UTC and calculate available durations; preserve unavailable timing explicitly. | +| Inputs, outputs, errors, nested JSON | Recursively redact credentials and known secret literals before persistence. | +| Authorization headers and token payloads | Omit or mask; protected runtime payloads stay protected. | +| URL query values | Omit credential-bearing query content. | +| Source VERSION | Record value and hash separately from the version observed in run output. | +| Parent/child evidence | Keep exact verified associations; label weaker correlation claims. | +| Hidden runtime/model settings | Null plus a reason, not guessed values. | +| Typed null, empty string, false, zero | Preserve collected values rather than treating false-like values as missing. | + +Azure runtime status, semantic test result, authentication result, evidence +completeness, and gallery readiness are different facts. A successful export +does not establish readiness; a complete report can legitimately be NotReady. + +The default `FailureFocused` scope preserves every available action and +repetition's metadata, timing, status, errors, and retries. It collects +top-level payloads, failed/timed-out/cancelled repetitions, successful +semantic-result producers, and child-workflow calls. Repetitive successful +poll payloads are explicitly `NotCollectedByScope`, not failed downloads. +Completeness is defined within that declared scope. + +## Collection and recovery lessons + +| Observed issue in earlier combined development | Reporting response | +|---|---| +| Long exports outlived cached ARM credentials | Refresh by expiry and perform one bounded authentication retry. | +| Until repetition routes returned 404 | Collect supported nested-action repetitions and container metadata instead. | +| Repeated checkpoint sanitization dominated execution | Sanitize incrementally and batch checkpoint saves. | +| Successful polling payloads caused excessive requests | Preserve metadata without downloading redundant successful poll payloads. | +| Valid empty Compose outputs appeared missing | Preserve typed values and require corroboration for any legacy offline correction. | +| `code` masking also matched diagnostic status/error labels | Bound OAuth-code matching while preserving diagnostic status codes. | +| Metadata-only refresh repeated an expensive ARM pass | Reuse a complete immutable report and supported offline writer/event functions. | +| Concurrent workers replaced cleanup inventories | One writer per run folder; preserve snapshots, scope, and count differences instead of inventing lost history. | + +An empty JSON string is present evidence, not evidence that a download +failed. A protected token is a security exclusion, not a request to remove +security settings. Neither correction changes functional test results. + +## Verification and limits + +Run: + +```powershell +.\Test-Authentication.ps1 +.\Test-ScimValidationReport.ps1 +``` + +The reporter tests use synthetic fixtures and no Azure calls. A fixture +VERSION or historical deployed version is not the current bundle version. +Validate source/version equality, LF bytes, links, exact-run association, +known-secret literals, encoded Basic credentials, JWTs, and archive hashes. +Synthetic token fixtures are allowed in test source, never in shareable +reports. + +Earlier combined development produced a complete OAuth-only report, but +does not constitute a version-15 live run or certification of this separated +branch. Reuse sanitized historical evidence only with exact version/hash +attribution. Do not start a concurrent validation against the authentication +branch's endpoint or modify its apps, jobs, artifacts, or source. + +The [canonical agent](scim-onboarding.agent.md), [setup walkthrough](SetupLogicApp-Standard-Agent.md), +and [test overview](SCIM-Validation-Test-Overview.md) contain the reporting +integration. Fresh authentication validation belongs to the higher-priority +authentication workstream; reporting must not delay or contaminate it. diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/scim-onboarding.agent.md b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/scim-onboarding.agent.md index 652b4d9c..b3021504 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/scim-onboarding.agent.md +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/scim-onboarding.agent.md @@ -22,7 +22,25 @@ description: > # SCIM Onboarding Validation Agent -The same-bundle [VERSION](VERSION) file is the sole source of the bundle version. Read it before deployment and synchronize `LogicAppVersion`. Use the canonical agent, parameters, and all seven workflow definitions from one source revision. Do not overwrite a locally supplied authentication-aware bundle with older downloads. +The same-bundle [VERSION](VERSION) file is the sole source of truth for the overall bundle, workflows, and agent version. Read it at startup; do not maintain a separate agent revision. Every change to this bundle must bump `VERSION` and synchronize the `LogicAppVersion` parameter. Record the version value, VERSION file SHA-256, agent SHA-256, source revision, and actual deployed version in every run report. Stop before deployment if the template version differs from `VERSION`; never label a historical run with a newer source version. + +Record the UTC agent-run start and starting current directory before the first operation. Immediately read the same-bundle `VERSION` and create a unique run subfolder directly under that starting directory, before asking for inputs or touching Azure: + +```powershell +$agentRunStartedUtc = [datetimeoffset]::UtcNow +$startingDirectory = (Get-Location).ProviderPath +$bundleVersion = (Get-Content -LiteralPath (Join-Path $sourceWorkflowDirectory 'VERSION') -Raw).Trim() +if ($bundleVersion -notmatch '^\d+\.\d+(?:\.\d+)?$') { throw 'The same-bundle VERSION is missing or invalid.' } +$localRunId = [guid]::NewGuid().ToString('N').Substring(0, 8) +$artifactPrefix = 'scimrun-v{0}-{1}-{2}' -f $bundleVersion, $agentRunStartedUtc.ToString('yyyyMMddTHHmmssZ'), $localRunId +$workDirectory = Join-Path $startingDirectory $artifactPrefix +$null = New-Item -ItemType Directory -Path $workDirectory -ErrorAction Stop +``` + +Use the equivalent operations on other operating systems. If source files must first be resolved, capture the starting directory/time first, resolve `VERSION` from the exact chosen source revision, then create the subfolder before validation. Do not use a machine-specific path, a parent/sibling folder, or an independently supplied work-directory root. Keep full endpoint and Azure identifiers in metadata, not in the folder name. + +Keep a sanitized event list with actual phase/step start and end times, status, and parent step, saved as `agent-events.json` in this folder. Do not record credentials or tokens in events. On a setup blocker, export Step 6g before stopping; after cleanup, finalize the agent-run end time and regenerate the report. Never invent timestamps that were not recorded. + You are the **SCIM onboarding validation agent**. You help ISVs validate that their SCIM provisioning integration is ready to publish to the Microsoft Entra app gallery. You convert a 28-step, 3-portal manual process into a guided conversational experience. You create all required Azure and Entra resources, deploy the validation Logic App, execute tests, diagnose failures, apply fixes, and re-run until the ISV has a clean validation outcome and is ready to follow the official submission process. @@ -96,7 +114,7 @@ Collect the ISV's SCIM endpoint and available authentication methods, validate t - `federatedAudience` (required for Google STS/SA flow, optional otherwise) - Otherwise set all federated fields to empty strings. The federation test must be Not applicable and must issue no Entra, Google, or ISV token requests. - e. **Run work directory** — accept an explicit working directory or ask where to save this validation run. Keep runs separate, retain the exact Azure run ID, and never persist credentials, tokens, raw parameters, or app settings. + e. **Run work directory** — use the versioned subfolder already created under the starting current directory. Record the local run ID now and the exact Azure run ID after triggering. Save sanitized source hashes and run artifacts there. Never persist credentials, tokens, raw parameters, or app settings. **Use `ask_user` for missing applicable inputs. Explicit initial values count as answers. Do not ask OAuth questions in a WIF-only environment, or federation questions in an OAuth-only environment. Missing credentials for a declared method are a configuration error, not permission to silently mark that method unavailable.** @@ -464,7 +482,7 @@ The Logic App has **7 workflows + 1 parameters file**. `Authentication_Workflow` **File acquisition (REQUIRED — use one coherent agent/template bundle):** -When this agent is supplied with local workflow templates and authentication tests, use that complete bundle. Never replace locally updated files with older downloads from a different revision. Record source revision, `VERSION`, dirty state, and file hashes. Otherwise download all files from the same repository revision; do not mix agent, parameter, workflow, and authentication-test versions. +When this agent is supplied with local workflow templates and report scripts, use that complete bundle. Never replace locally updated files with older downloads from a different revision. Record source revision, `VERSION`, dirty state, and file hashes. Otherwise download all files from the same repository revision; do not mix agent, parameter, workflow, and report-script versions. The canonical source of truth for the template files is: @@ -472,7 +490,7 @@ The canonical source of truth for the template files is: Raw download base: `https://raw.githubusercontent.com/AzureAD/SCIMReferenceCode/master/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/` -Resolve one complete local bundle or one immutable Git commit first. The deployment set contains seven workflows and one parameters template; retain `VERSION`, the canonical agent and authentication tests alongside them. Do not fall back file-by-file to a different revision. If any required file is missing, abort Phase 2 and report the missing file. +Resolve one complete local bundle or one immutable Git commit first. The deployment set contains seven workflows and one parameters template; retain `VERSION`, the canonical agent, exporter, and test scripts alongside them. Do not fall back file-by-file to a different revision. If any required file is missing, abort Phase 2 and report the missing file. Files to acquire (exact names, case-sensitive): @@ -493,7 +511,7 @@ Example acquisition loop the agent must follow before any upload: # Use this acquisition path only after choosing GitHub, in an empty source subfolder of the run directory. # SOURCE_REVISION is the selected immutable commit, not a moving branch name. BASE_RAW="https://raw.githubusercontent.com/AzureAD/SCIMReferenceCode/$SOURCE_REVISION/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp" -FILES=(Orchestrator_Workflow.json Initialization_Workflow.json UserTests_Workflow.json GroupTests_Workflow.json SCIMTests_Workflow.json SCIMToken_Workflow.json Authentication_Workflow.json Orchestrator_Parameters.json VERSION scim-onboarding.agent.md Test-Authentication.ps1) +FILES=(Orchestrator_Workflow.json Initialization_Workflow.json UserTests_Workflow.json GroupTests_Workflow.json SCIMTests_Workflow.json SCIMToken_Workflow.json Authentication_Workflow.json Orchestrator_Parameters.json VERSION scim-onboarding.agent.md Export-ScimValidationReport.ps1 Test-Authentication.ps1 Test-ScimValidationReport.ps1) for f in "${FILES[@]}"; do if curl -fsSL "$BASE_RAW/$f" -o "$f.download"; then @@ -1531,6 +1549,76 @@ while (test run fails): --- +### Step 6g: Export a shareable run report (REQUIRED for every outcome) + +Run this step before cleanup or submission, including failed, cancelled, timed-out, and setup-blocked outcomes. Download `Export-ScimValidationReport.ps1` from the **same revision/bundle** as this agent. Do not substitute the latest run for the exact captured run ID. + +Create a **sanitized** `run-metadata.json` in the work directory. Use these optional keys: +- `environment`: tenant/subscription/resource group/Logic App identifiers; identifier-only `managedIdentity`, `entraApplication`, `servicePrincipal`, `provisioningJob`, `endpoint`, and `operatorIdentity` objects. +- `supportedAuthenticationMethods` and `scimAuthenticationMethod`: confirmed declarations and direct-request transport. +- `source`: repository URL, revision, branch, dirty state, and description. The exporter records observed Git/file provenance separately from these caller claims. +- Version provenance: the exporter reads `VERSION` from `WorkflowDirectory`. Record the actual deployed `LogicAppVersion` and `authenticationPolicy` from this exact run separately from current source provenance. Version or source mismatches must remain visible; never relabel older evidence. +- `deploymentOverlay`: only when an overlay was intentionally deployed; record its name, description, revision/hash, and deployed workflow hashes separately from the canonical source. Never claim source/deployment equivalence without verification. +- `harness`: publicly exposed name/version/model/reasoning/context information. Omitted or unavailable values remain null with an explicit reason. Never invent a model, thinking level, context size, or hidden runtime state. +- `preflight`: for setup-blocked runs, use `{ "status": "Blocked", "reason": "", "startTime": "", "endTime": "" }`. +- `agentRun`: actual setup-to-cleanup UTC `startTime`, `endTime`, and `status`, separate from the Azure Orchestrator window. Use null with a reason when timing was not recorded; never reconstruct exact timestamps from memory. +- `agentEventsPath`: relative path to the recorded event array. Each event has `name`, `kind`, `parentName`, `startTime`, `endTime`, `status`, and sanitized `details`. Record phase, step, credential-validation, deployment, and cleanup boundaries as they occur. In-progress events have a null end time. +- `workflowRuns`: optional exact observed child IDs and parent-action/repetition relationships when secured call outputs hide the run-ID header. Do not guess these relationships. + +Keep the metadata limited to identifiers and sanitized context. **Never use raw parameters, app settings, connectivity credentials, tokens, or authorization headers as metadata.** + +```powershell +$reportArguments = @{ + WorkDirectory = $workDirectory + SubscriptionId = $subscriptionId + TenantId = $tenantId + ResourceGroupName = $resourceGroupName + LogicAppName = $logicAppName + OrchestratorRunId = $runId + AgentPath = $canonicalAgentPath + WorkflowDirectory = $sourceWorkflowDirectory + MetadataPath = Join-Path $workDirectory 'run-metadata.json' + Zip = $true +} +& $reportExporter @reportArguments -KnownSecrets $knownSecretsHeldInMemory +$reportExitCode = $LASTEXITCODE +``` + +For preflight-blocked runs, omit `OrchestratorRunId`; never invent a run ID. Pass known credential literals through a caller-held in-memory array for defense-in-depth redaction, not command-line literals or files. + +The exporter produces **`validation-run-details.json`**, **`validation-report.md`**, and, with `-Zip`, a portable report archive. These stable filenames are working aliases. After export, also create versioned copies using the startup timestamp, bundle version, and actual Azure run ID (or the local run ID for a setup blocker): + +```powershell +$executionId = if ($runId) { $runId } else { $localRunId } +$exportPrefix = 'scimrun-v{0}-{1}-{2}' -f $bundleVersion, $agentRunStartedUtc.ToString('yyyyMMddTHHmmssZ'), $executionId +foreach ($suffix in @('validation-run-details.json', 'validation-report.md', 'validation-report.zip')) { + $path = Join-Path $workDirectory $suffix + if (Test-Path -LiteralPath $path) { + Copy-Item -LiteralPath $path -Destination (Join-Path $workDirectory "$exportPrefix-$suffix") -Force + } +} +``` + +The run folder namespaces all other files; keep the short stable names required by helpers within it. Generate the final report only after all child workflows are terminal. Run only one collector per work directory. Before resuming, verify that the prior process has actually stopped; inspect checkpoint timestamps and counts rather than assuming a quiet process is inactive. For an interrupted collection of the same terminal run, omit `-ResumeReportPath` to automatically resume its matching checkpoint. Use `-ResumeReportPath` when explicitly importing a prior final report; successful immutable payloads can be reused without recollection. + +Reports record run/workflow/test/action/repetition timing, retries, nested errors, sanitized request/response URLs/headers/bodies, environment and source/version provenance, and explicit evidence gaps. Protected token payloads are intentional security exclusions; never disable secure inputs/outputs to collect them. Pagination and nested authentication/token workflow invocations must be included. Never call an unsupported Until `/repetitions` endpoint; collect the container timing/counts and supported nested action repetitions instead. + +The exporter declares a `FailureFocused` diagnostic scope: preserve all action/repetition metadata without sampling, collect top-level payloads and failed/timed-out/cancelled/aborted repetition payloads, and retain successful semantic-result and child-workflow-call payloads. Successful repeated polling payloads are marked `NotCollectedByScope`; omitting them is intentional, not a download failure. Evidence completeness applies to this declared scope and must never be described as collection of every successful polling payload. + +After a complete exact-run export, a change only to recorded cleanup/lifecycle metadata does **not** require another ARM history pass. Archive the complete report and refresh that caller-supplied metadata offline using the same-bundle exporter functions: `Get-ScimAgentEvents`, `Get-ScimRecordedTiming`, `Get-ScimSanitizedReportSnapshot`, and `Write-ScimValidationReport`. Preserve all test outcomes, exact-run associations, executed/source provenance, immutable action evidence, and genuine collection failures. Replace only obsolete optional diagnostics for the metadata actually refreshed, and recompute the optional-gap count. Record the original report and finalizer hashes and what changed. Write a separate final directory, verify it, then refresh stable/versioned aliases. + +A separately collected provisioning audit may be attached only with explicit job/application/time scope and redaction; do not imply per-object correlation that was not verified. Retain earlier conservative masks rather than guessing their original values. If an unnecessary refresh attempt is interrupted, preserve its checkpoint and failure evidence, clearly record why complete original evidence remains authoritative, and do not substitute the incomplete attempt or leave it eligible for accidental automatic resumption. If the original export was incomplete or actual Azure evidence changed, use normal collection/resumption instead of this metadata-only path. + +Preserve collected JSON null, empty strings, false, and zero as values; do not classify them as missing merely because they are false-like. If an older exporter misclassified a successful Compose's collected empty string as missing, archive the original report before using the narrowly guarded offline correction below. It makes no Azure requests, requires evidence supporting each correction, records the original report and corrector hashes under `offlineFinalization`, and preserves historical versions, test results, and real evidence gaps. Write to a separate directory; an evidence-complete report can still be NotReady because tests failed. + +```powershell +. "\Export-ScimValidationReport.ps1" +$report = Complete-ScimValidationReport -ReportPath "" +Write-ScimValidationReport $report "" -Zip +``` + +Exit codes: **0** = complete and ready; **1** = complete but not ready; **2** = incomplete or blocked. A successful export is not itself a passing validation. Review the report and its completeness warnings, verify that no credential is present, and record both the runtime status and semantic readiness. Reports still contain organizational identifiers and test-user data; have the ISV review them before sharing. Export again after any approved rerun using that rerun's exact ID. + ## Submit Test Results (Official Guide) ### Goal @@ -1546,6 +1634,7 @@ Give the ISV a complete, copy-ready submission summary for the exact successful - Require each declared `Validate_Credentials_Test` and `Federated_Identity_Test` result to pass independently. Exclude only undeclared Not applicable methods. Require the static-token connectivity result too when `Bearer` was selected. No selected failure may be hidden by another method's success. - `optionalSuccessTestsResult == "PASSED"` is preferred; `"WARNING"` is acceptable — call out each warning test explicitly in the summary. - No Phase 6 issue remains unresolved. + - Step 6g produced a complete report for this exact run with no unresolved evidence gaps. Do not print submission details for a failed, cancelled, timed-out, running, or stale run. A workflow status of `Succeeded` and the test-result totals are separate facts; verify and print both. From 7696b0d011813e4e735eb794e994dc7b85a959cd Mon Sep 17 00:00:00 2001 From: "Prashant Rane (ALLYIS INC)" Date: Fri, 16 Oct 2026 00:43:24 -0600 Subject: [PATCH 2/2] Preserve report array shapes and embedded credential redaction Count protected repetition payloads in security exclusions and cover the reproduced defects with offline regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../Export-ScimValidationReport.ps1 | 30 +++++++++++----- .../Test-ScimValidationReport.ps1 | 34 ++++++++++++++++++- .../Validation-Reporting-Changes.md | 9 +++++ 3 files changed, 64 insertions(+), 9 deletions(-) diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 index 1fe969cb..03b49990 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Export-ScimValidationReport.ps1 @@ -166,23 +166,24 @@ function Test-ScimSensitiveKey { } function Protect-ScimValue { - param($Value, [string[]] $Secrets = @(), [string] $Name = '', [int] $Depth = 0) + param($Value, [string[]] $Secrets = @(), [string] $Name = '', [int] $Depth = 0, [switch] $RedactCode) if ($Depth -gt 80) { return '[OMITTED: depth limit]' } - if ((Test-ScimSensitiveKey $Name) -and $Name -ne 'rawCredentialsPersisted') { return '[REDACTED]' } + if (((Test-ScimSensitiveKey $Name) -and $Name -ne 'rawCredentialsPersisted') -or ($RedactCode -and $Name -eq 'code')) { return '[REDACTED]' } if ($null -eq $Value) { return $null } if ($Value -is [string]) { $s = Protect-ScimText $Value $Secrets $trimmed = $Value.Trim() if ($trimmed.StartsWith('{') -or $trimmed.StartsWith('[') -or $trimmed.StartsWith('"')) { try { - $embedded = ConvertFrom-Json -InputObject $Value -AsHashtable -ErrorAction Stop - return (Protect-ScimValue $embedded $Secrets '' ($Depth + 1) | ConvertTo-Json -Depth 100 -Compress) + $embedded = ConvertFrom-Json -InputObject $Value -AsHashtable -NoEnumerate -ErrorAction Stop + # Parsing embedded text must not bypass its conservative OAuth-code masking. + return (Protect-ScimValue $embedded $Secrets '' ($Depth + 1) -RedactCode | ConvertTo-Json -Depth 100 -Compress) } catch { } } # Encoded form bodies and error strings often carry JSON or secrets one layer down. if ($s -match '%[0-9a-fA-F]{2}') { $decoded = [uri]::UnescapeDataString($s) - if ($decoded -ne $s) { return Protect-ScimValue $decoded $Secrets '' ($Depth + 1) } + if ($decoded -ne $s) { return Protect-ScimValue $decoded $Secrets '' ($Depth + 1) -RedactCode:$RedactCode } } return $s } @@ -196,13 +197,13 @@ function Protect-ScimValue { if ($Name -match '(?i)headers$' -and $k -notmatch '^(?i:content-type|content-length|date|retry-after|x-ms-request-id|x-ms-correlation-request-id|x-ms-client-tracking-id|x-ms-workflow-run-id|traceparent)$') { $result[$safeKey] = '[REDACTED: non-allowlisted header]' } else { - $result[$safeKey] = Protect-ScimValue $Value[$k] $Secrets ([string]$k) ($Depth + 1) + $result[$safeKey] = Protect-ScimValue $Value[$k] $Secrets ([string]$k) ($Depth + 1) -RedactCode:$RedactCode } } return $result } if ($Value -is [System.Collections.IEnumerable]) { - return ,@($Value | ForEach-Object { Protect-ScimValue $_ $Secrets '' ($Depth + 1) }) + return ,@($Value | ForEach-Object { Protect-ScimValue $_ $Secrets '' ($Depth + 1) -RedactCode:$RedactCode }) } return $Value } @@ -641,6 +642,16 @@ function Get-ScimInvocationEvidence { } } +function Get-ScimActionSecurityExclusionCount { + param($Entries) + $count = 0 + foreach ($entry in $Entries) { + $count += @($entry.securityExclusions | Where-Object { $null -ne $_ }).Count + $count += Get-ScimActionSecurityExclusionCount $entry.repetitions + } + return $count +} + function Select-ScimMetadataFields { param($Value, [string[]] $Names) $selected = @{} @@ -1104,7 +1115,10 @@ function New-ScimValidationReport { $report.evidenceFailures = @($ctx.failures) $missing = @($ctx.failures | Where-Object required).Count $report.securityExclusions = @($ctx.securityExclusions) - $securedPayloadCount = @($report.workflows | ForEach-Object { $_.actions | ForEach-Object { $_.securityExclusions } }).Count + $securedPayloadCount = 0 + foreach ($workflow in $report.workflows) { + $securedPayloadCount += Get-ScimActionSecurityExclusionCount (@($workflow.triggers) + @($workflow.actions)) + } $report.completeness = @{ complete = $missing -eq 0; requiredUnavailableCount = $missing optionalUnavailableCount = @($ctx.failures | Where-Object { -not $_.required }).Count diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 index 5207568d..ad681c7f 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Test-ScimValidationReport.ps1 @@ -56,6 +56,25 @@ try { Assert-Report ($safeJson.Contains('application/json')) 'Allowlisted content-type retained' Assert-Report ($safeJson.Contains('PASSED')) 'Semantic result retained' Assert-Report ((ConvertTo-ScimCanonicalJson (Protect-ScimValue $safe @($secret))) -eq $safeJson) 'Repeated sanitization preserves already-redacted fixture content' + foreach ($embeddedJson in @( + '[]', '[null]', '[false]', '[0]', '[""]', '[{"result":"FAILED"}]', + '[[],[false]]', '{"empty":[],"results":[{"result":"FAILED"}]}' + )) { + $embeddedSafe = Protect-ScimValue @{ body = $embeddedJson } + Assert-Report ($embeddedSafe.body -ceq $embeddedJson) "Embedded JSON retains array shape and typed values: $embeddedJson" + Assert-Report ((Protect-ScimValue $embeddedSafe).body -ceq $embeddedJson) "Repeated sanitization retains embedded JSON shape: $embeddedJson" + } + $embeddedSecret = Protect-ScimValue @{ body = '[{"result":"FAILED","password":"fixture-array-password"}]' } + Assert-Report ($embeddedSecret.body -ceq '[{"password":"[REDACTED]","result":"FAILED"}]') 'Singleton embedded JSON arrays retain both their shape and credential redaction' + Assert-Report ((Get-ScimActionSecurityExclusionCount @()) -eq 0) 'An empty action inventory has no security exclusions' + $nestedExclusions = @(@{ + securityExclusions = @(@{ source = 'inputs' }) + repetitions = @(@{ + securityExclusions = @(@{ source = 'outputs' }) + repetitions = @(@{ securityExclusions = @(@{ source = 'inputs' }, @{ source = 'outputs' }) }) + }) + }) + Assert-Report ((Get-ScimActionSecurityExclusionCount $nestedExclusions) -eq 4) 'Security exclusion counting retains multiple repetition levels without counting absent collections' $prose = 'Acquire an ARM token for the selected tenant.' Assert-Report ((Protect-ScimText $prose) -eq $prose) 'Harmless token-for prose is not treated as a credential assignment' $maskedUrl = 'https://sample.azure.com/path?[REDACTED QUERY]' @@ -90,6 +109,18 @@ try { $diagnosticLogs.provisioningLogs.errorCode -eq 'SystemForCrossDomainIdentityManagementServiceIncompatible') 'Nested provisioning diagnostics retain HTTP400, invalidSyntax and structured errorCode' Assert-Report (-not $diagnosticLogs.provisioningLogs.reason.Contains('fixture-oauth-code') -and -not $diagnosticLogs.provisioningLogs.reason.Contains('fixture-other-code')) 'OAuth code assignments are still redacted beside retained provisioning diagnostics' + foreach ($embeddedCode in @( + '{"code":"fixture-embedded-oauth-code","errorCode":"invalidSyntax","statusCode":400}', + '[{"code":"fixture-embedded-oauth-code","errorCode":"invalidSyntax","statusCode":400}]', + '{"outer":"{\"code\":\"fixture-embedded-oauth-code\",\"errorCode\":\"invalidSyntax\",\"statusCode\":400}"}' + )) { + $safeCode = Protect-ScimValue @{ body = $embeddedCode } + Assert-Report (-not $safeCode.body.Contains('fixture-embedded-oauth-code') -and $safeCode.body.Contains('[REDACTED]')) 'Embedded JSON cannot bypass textual OAuth-code redaction' + Assert-Report ($safeCode.body.Contains('invalidSyntax') -and $safeCode.body.Contains('400')) 'Embedded JSON retains diagnostic errorCode and statusCode values beside a redacted OAuth code' + Assert-Report ((Protect-ScimValue $safeCode).body -ceq $safeCode.body) 'Embedded OAuth-code redaction remains idempotent' + } + $structuredDiagnostic = Protect-ScimValue @{ error = @{ code = 'FixtureDiagnostic' }; statusCode = 400 } + Assert-Report ($structuredDiagnostic.error.code -ceq 'FixtureDiagnostic' -and $structuredDiagnostic.statusCode -eq 400) 'Typed runtime diagnostic codes remain intact outside embedded JSON text' Assert-Report ((Protect-ScimValue @{ message = 'ordinary failure'; result = 'FAILED' }).message -eq 'ordinary failure') 'Ordinary error retained' Assert-Report ((ConvertTo-ScimCanonicalJson @{ z = 2; a = @{ b = 3; a = 1 } }) -eq (ConvertTo-ScimCanonicalJson @{ a = @{ a = 1; b = 3 }; z = 2 })) 'Canonical key ordering' $authCalls = [System.Collections.Generic.List[object]]::new() @@ -474,7 +505,7 @@ try { # A self-contained source repository proves provenance without depending on parent edits. & git -C $fixtureDirectory init --quiet & git -C $fixtureDirectory add --all - & git -C $fixtureDirectory -c user.name=Fixture -c user.email=fixture@example.invalid -c commit.gpgsign=false commit --quiet -m 'Offline reporting fixture' + & git -C $fixtureDirectory -c maintenance.auto=false -c user.name=Fixture -c user.email=fixture@example.invalid -c commit.gpgsign=false commit --quiet -m 'Offline reporting fixture' if ($LASTEXITCODE -ne 0) { throw 'Fixture Git initialization failed.' } $state = @{ @@ -606,6 +637,7 @@ try { Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-1')[0].timing.durationMilliseconds -eq 2000) 'Nested child timing collected' Assert-Report (@($report.workflows | Where-Object runId -eq 'exact-auth-1')[0].run.properties.error.code -eq 'RetryObserved') 'Nested child error retained safely' Assert-Report ($report.completeness.securityExcludedCount -gt 0 -and $report.securityExclusions.Count -eq 1) 'Secured child evidence is excluded, not falsely incomplete' + Assert-Report ($report.completeness.securityExcludedCount -eq 15) 'Security exclusion total includes ten top-level payload exclusions, four protected repetition payloads and one hidden child mapping' Assert-Report ($report.suppliedSource.metadata.revision -eq 'caller-source-revision' -and $report.provenance.sourceGit.revision -ne 'caller-source-revision') 'Caller source claims do not override observed source revision' Assert-Report ($report.deploymentOverlay.metadata.workflowHashes[0].sha256 -eq ('b' * 64)) 'Allowlisted deployment-overlay hashes retained' Assert-Report ($report.provenance.agent.sha256 -eq (Get-FileHash $agentPath).Hash.ToLowerInvariant()) 'Agent SHA256' diff --git a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md index 50924955..b2048da2 100644 --- a/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md +++ b/Microsoft.SCIM.LogicAppValidationTemplate/StandardLogicApp/Validation-Reporting-Changes.md @@ -61,6 +61,9 @@ protection, or turn missing required evidence into success. | Parent/child evidence | Keep exact verified associations; label weaker correlation claims. | | Hidden runtime/model settings | Null plus a reason, not guessed values. | | Typed null, empty string, false, zero | Preserve collected values rather than treating false-like values as missing. | +| JSON arrays embedded in strings | Preserve empty/singleton arrays and nested typed values while recursively redacting their contents. | +| OAuth `code` assignments in embedded JSON text | Apply the same conservative masking as textual logs, without matching diagnostic `statusCode` or `errorCode` fields. Typed runtime diagnostic codes remain intact. | +| Protected action/repetition payloads | Count recorded security exclusions across triggers, actions, and every repetition level, separately from hidden child mappings and evidence gaps. | Azure runtime status, semantic test result, authentication result, evidence completeness, and gallery readiness are different facts. A successful export @@ -90,6 +93,12 @@ An empty JSON string is present evidence, not evidence that a download failed. A protected token is a security exclusion, not a request to remove security settings. Neither correction changes functional test results. +Offline regression review also corrected embedded JSON array unwrapping, +an embedded-JSON bypass of textual OAuth-code masking, and an exclusion-summary +undercount that omitted protected repetitions. These fixes affect newly +generated reports only; frozen reports and their original counts, hashes, +versions, and outcomes must not be rewritten. + ## Verification and limits Run: