Skip to content

feat(github): add bounded evidence transport and rate limits - #2633

Merged
merill merged 3 commits into
maester365:mainfrom
thetechgy:feat/github-evidence-transport
Nov 4, 2026
Merged

merill merged 3 commits into
maester365:mainfrom
thetechgy:feat/github-evidence-transport

Conversation

@thetechgy

@thetechgy thetechgy commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

This separates the shared GitHub evidence handling from #2525 so it can be reviewed independently of the deletion assessments.

The changes cover the following:

  • For REST evidence: Add optional response metadata, bounded pagination, collection-integrity information, and version-qualified cache entries. Existing callers still receive the response body unless they request metadata.
  • For GraphQL evidence: Add a read-only query transport with separate cache entries and response/error metadata. The GraphQL helper has no production caller in this PR; its first consumer is introduced in fix(github): assess deletion access with configured approvals #2635.
  • For rate limits and timeouts: Keep REST and GraphQL primary quotas separate and share secondary-limit protection across the connection, including responses that signal both limits. Each limit retains its own deadline. Requests use 30-second connection/header and stalled-body limits, without automatic retries. These are not overall assessment deadlines.
  • For repository inventory: Collect and validate repository identities without filtering out archived repositories. A successful inventory read does not establish that the caller can see the whole organization.
  • For affected objects: Recognize supported legacy and metadata REST cache keys without interpreting opaque GraphQL entries as resource URLs.

Evidence remains a session snapshot until a new run, reconnect/disconnect or module reset clears it. Standalone Invoke-MtTest calls reuse that snapshot. The GraphQL transport is added to the existing cache-switch allowlist; checks do not bypass the cache.

Validation

On Linux / PowerShell 7.6.6 / Pester 5.7.1, the full unit-test command completed with 5 failures out of 6,691 tests. All five reproduced on the unchanged PR head 9e921e48b2274799d004fac5c8ef691220f3794e: one console-colour assertion and four Azure request URI assertions. All 79 console tests passed in a separate run with process-local colour settings normalized. These baseline failures were not changed in this PR.

The GitHub transport tests, including real loopback header/body timeout fixtures, passed in the full run. All 34 focused rate-limit tests passed, including 12 combined-limit cases covering REST HTTP errors, GraphQL HTTP errors and HTTP 200 GraphQL errors. The new combined-limit cases failed before the fix. Changed-file PSScriptAnalyzer, module build and output validation passed. Read-only reports from a combined build exercised normal REST and GraphQL collection against a live EMU organization. The reports do not contain a source revision, so they do not establish exact-revision validation of this PR. Live timeout, rate-limit and GHE.com behavior remains unverified; those failure paths are covered by the fixtures. No tenant resources were changed.

Dependencies

This PR is independent of the setting fixes in #2525 and reporting in #2634. Those three may merge independently; merging #2525 first is recommended. #2635 requires both this PR and reporting to merge first.

PR #2540 has reconciled this transport and the other foundations at e70990fab7c2a2dad034b9840502d89d59d7eab5. After the prerequisites merge, reconcile the dependent branches against actual upstream main and repeat affected validation.

Current revision: 4ba540611848b749b93cd547173eb4dd2f1a4ff6. Hosted validation is running for this revision. Maintainer review and approval remain required.

Summary by CodeRabbit

  • New Features
    • GitHub repository inventories can now be collected with duplicate and conflicting entries identified.
    • GitHub REST and GraphQL reads provide clearer pagination and collection-integrity information.
    • GitHub GraphQL read results can be reused from cache when caching is enabled.
  • Bug Fixes
    • GitHub rate limits are recognized across REST and GraphQL requests, with cooldowns respected across requests.
    • Slow GitHub requests now have bounded timeouts, and incomplete paginated results are not cached.
    • Affected-object cache extraction now supports additional GitHub cache formats and skips invalid URLs.

@coderabbitai

coderabbitai Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 9e921e4 and 4ba5406.

Files selected for processing: 7
  • powershell/internal/services/github/Get-MtGitHubRateLimit.ps1
  • powershell/internal/services/github/Get-MtGitHubRateLimitMessage.ps1
  • powershell/internal/services/github/Invoke-MtGitHubRateLimitGuard.ps1
  • powershell/tests/functions/Get-MtGitHubRepositoryInventory.Tests.ps1
  • powershell/tests/functions/GitHubAffectedObjects.Tests.ps1
  • powershell/tests/functions/GitHubRateLimit.Tests.ps1
  • powershell/tests/functions/GitHubTransportTimeout.Tests.ps1
Files skipped from review as they are similar to previous changes: 3
  • powershell/tests/functions/GitHubAffectedObjects.Tests.ps1
  • powershell/tests/functions/Get-MtGitHubRepositoryInventory.Tests.ps1
  • powershell/tests/functions/GitHubTransportTimeout.Tests.ps1

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Walkthrough

The PR adds GitHub REST metadata requests, GraphQL read requests, and connection-scoped rate-limit tracking. It adds repository inventory validation, tightens affected-object cache-key parsing, and adds Pester coverage for these behaviors.

Changes: GitHub data collection

Layer / File(s) Summary
Rate-limit classification and connection state
powershell/internal/services/github/Get-MtGitHubErrorStatusCode.ps1, powershell/internal/services/github/Get-MtGitHubRateLimit.ps1, powershell/internal/services/github/Get-MtGitHubRateLimitMessage.ps1, powershell/internal/services/github/Invoke-MtGitHubRateLimitGuard.ps1, powershell/internal/session/Clear-ModuleVariable.ps1, powershell/public/connect/Connect-MtGitHub.ps1, powershell/public/connect/Disconnect-MtGitHub.ps1, powershell/tests/functions/GitHubRateLimit.Tests.ps1
Rate-limit classification returns primary and secondary limit records. The guard tracks limits by connection, authority, and scope. Connection probes use the guard, and disconnect clears rate-limit state. Tests cover scopes, cooldowns, pagination, and connection probes.
REST metadata and pagination
powershell/internal/services/github/Invoke-MtGitHubRequest.ps1, powershell/tests/functions/Invoke-MtGitHubRequest.Tests.ps1, powershell/tests/functions/GitHubTransportTimeout.Tests.ps1
REST requests add metadata, per-call API-version overrides, pagination validation, partial-result evidence, and 30-second timeouts. Tests cover pagination, cache behavior, and stalled REST and GraphQL requests.
Repository inventory validation
powershell/internal/services/github/Get-MtGitHubRepositoryInventory.ps1, powershell/tests/functions/Get-MtGitHubRepositoryInventory.Tests.ps1
The inventory function validates collection completeness and repository identities, reports issues, and excludes conflicting identities. Tests cover duplicate and conflicting records, invalid evidence, and empty inventories.
GraphQL read requests
powershell/internal/services/github/Invoke-MtGitHubGraphQLRequest.ps1, powershell/tests/functions/Invoke-MtGitHubGraphQLRequest.Tests.ps1, powershell/tests/general/GraphCacheUsage.Tests.ps1
The GraphQL request function validates read queries and responses, applies rate-limit checks and timeouts, and caches eligible results. Tests cover request construction, cache behavior, errors, and connection changes.
Affected-object cache extraction
powershell/internal/report/affected-objects/Get-MtAffectedObjectFromCache.ps1, powershell/tests/functions/GitHubAffectedObjects.Tests.ps1
Affected-object extraction accepts specified legacy and metadata REST cache keys and rejects malformed or unrelated keys. Tests cover extraction and deduplication.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Invoke-MtGitHubRequest
  participant Invoke-MtGitHubRateLimitGuard
  participant GitHubRESTAPI
  participant SessionCache
  Invoke-MtGitHubRequest->>Invoke-MtGitHubRateLimitGuard: Check quota before request
  Invoke-MtGitHubRateLimitGuard-->>Invoke-MtGitHubRequest: Allow or block request
  Invoke-MtGitHubRequest->>GitHubRESTAPI: Send bounded request
  GitHubRESTAPI-->>Invoke-MtGitHubRequest: Return response and headers
  Invoke-MtGitHubRequest->>Invoke-MtGitHubRateLimitGuard: Record response quota
  Invoke-MtGitHubRequest->>SessionCache: Cache complete, integrity-valid result
Loading

Possibly related issues

  • maester365/maester#2524 — The PR tests clearing GraphQL cache evidence on disconnect and reconnect, which relates to stale cached data after a tenant change.

Possibly related PRs

  • maester365/maester#2635 — This PR consumes the REST and GraphQL transports, inventory results, pagination, and rate-limit handling added here.
  • maester365/maester#2525 — This PR consumes the repository inventory, REST pagination, GraphQL reads, and rate-limit handling provided here.
  • maester365/maester#2646 — This PR renders and navigates affected-object records whose GitHub extraction changes are included here.

Suggested labels: enhancement, status: needs review

Suggested reviewers: merill

Merge Risk: Minimal · up to 4ba54

This change adds bounded GitHub REST and GraphQL transport with shared rate-limit handling and repository inventory validation. No actionable merge-blocking risk remains in the reviewed changes. Maintainer review and approval are still required.

Pre-merge checks: Passed 5
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: bounded GitHub evidence transport and rate-limit handling.
Description check Passed The description is detailed and covers the implementation scope, validation results, dependencies, limitations, and testing status. It does not include the template's issue-closing entry or checkbox f…
Docstring Coverage Passed No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is detailed and covers the implementation scope, validation results, dependencies, limitations, and testing status. It does not include the template's issue-closing entry or checkbox format, but the required technical information is mostly complete.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (7 skipped: 7 unsupported.)

Finishing Touches
Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@thetechgy
thetechgy marked this pull request as ready for review November 2, 2026 17:59
@thetechgy
thetechgy requested a review from a team as a code owner November 2, 2026 17:59
Copilot AI balanced review requested due to automatic review settings November 2, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Rate-limit handling conflates API-specific primary resets with shared secondary cooldowns, potentially blocking other APIs unnecessarily or allowing requests before the secondary cooldown expires.

1 open issue
What changed in this PR

This PR adds shared GitHub evidence transport for Maester, independently of the deletion assessments that will consume it.

Changes:

  • Adds opt-in REST evidence metadata, bounded pagination, and a separately cached read-only GraphQL transport.
  • Introduces connection-scoped rate-limit protection and request timeouts.
  • Adds repository identity validation, REST cache-key recognition for affected objects, and focused regression tests.
File Description
powershell/​tests/​general/​GraphCacheUsage.Tests.ps1 Allows the GraphQL transport cache switch.
powershell/​tests/​functions/​Invoke-MtGitHubRequest.Tests.ps1 Tests REST metadata, pagination, caching, and timeouts.
powershell/​tests/​functions/​Invoke-MtGitHubGraphQLRequest.Tests.ps1 Tests GraphQL requests, errors, and cache isolation.
powershell/​tests/​functions/​GitHubTransportTimeout.Tests.ps1 Adds loopback fixtures for stalled responses.
powershell/​tests/​functions/​GitHubRateLimit.Tests.ps1 Tests cooldowns, quota separation, and bootstrap limits.
powershell/​tests/​functions/​GitHubAffectedObjects.Tests.ps1 Tests supported REST cache-key recognition.
powershell/​tests/​functions/​Get-MtGitHubRepositoryInventory.Tests.ps1 Tests repository identity validation and conflicts.
powershell/​public/​connect/​Disconnect-MtGitHub.ps1 Clears connection rate-limit state.
powershell/​public/​connect/​Connect-MtGitHub.ps1 Applies shared rate-limit protection to connection probes.
powershell/​internal/​session/​Clear-ModuleVariable.ps1 Documents cooldown preservation across cache resets.
powershell/​internal/​services/​github/​Invoke-MtGitHubRequest.ps1 Adds REST evidence metadata, pagination bounds, and timeouts.
powershell/​internal/​services/​github/​Invoke-MtGitHubRateLimitGuard.ps1 Records and enforces connection-scoped cooldowns.
powershell/​internal/​services/​github/​Invoke-MtGitHubGraphQLRequest.ps1 Adds cached read-only GraphQL transport.
powershell/​internal/​services/​github/​Get-MtGitHubRepositoryInventory.ps1 Collects repositories and validates their identities.
powershell/​internal/​services/​github/​Get-MtGitHubRateLimitMessage.ps1 Delegates diagnostics to shared limit classification.
powershell/​internal/​services/​github/​Get-MtGitHubRateLimit.ps1 Classifies limits and calculates cooldown deadlines.
powershell/​internal/​services/​github/​Get-MtGitHubErrorStatusCode.ps1 Reads status from structured rate-limit errors.
powershell/​internal/​report/​affected-objects/​Get-MtAffectedObjectFromCache.ps1 Recognizes REST metadata keys and excludes GraphQL keys.

Review effort: Balanced

@coderabbitai coderabbitai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Fix comments: Fix CodeRabbit comments on this PR

@merill merill left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @thetechgy, looks good to me. A couple of optional things if you get a chance: a line in the description saying the GraphQL helper has no caller until #2635, and the nested Invoke-ConnectionProbe in Connect-MtGitHub could move into its own internal helper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants