Skip to content

fix: Bound memory usage and in memory stats, improved hung worker cancellation - #73

Merged
Zacgoose merged 19 commits into
mainfrom
dev
Nov 12, 2026
Merged

Zacgoose merged 19 commits into
mainfrom
dev

Conversation

@Zacgoose

Copy link
Copy Markdown
Contributor

This pull request introduces several improvements and enhancements to resource tracking, logging, and retention diagnostics across the codebase. The main changes include adding new metrics and retention count reporting, enhancing log queue management to prevent memory growth under heavy load, and updating dependencies in CI workflows for improved reliability.

Resource and Retention Diagnostics

  • Added GetRetentionCounts() to WorkerMetricsBridge to collect and report counts of retained objects across major process-lifetime holders, including jobs, orchestrators, cache, real-time services, egress, HTTP diagnostics, queue labels, log queue depth, and dynamic assemblies. This helps spot potential leaks or unbounded growth.
  • Implemented CountRetained() in EgressLedger and HttpDiagnosticListener to support detailed retention reporting. [1] [2]
  • Exposed QueueDepth in LogBridge and LabelCount in QueueStatusBridge for inclusion in retention counts. [1] [2]

Logging Improvements

  • Changed FileLoggerProvider from an unbounded to a bounded channel (QueueCapacity = 50,000) for log entries. When full, the newest log lines are dropped and a warning is logged once the consumer catches up, preventing unbounded memory usage under log floods. [1] [2] [3] [4]

Queue and Metadata Management

  • QueueStatusBridge now tracks registration time for labels and trims old or excess labels, ensuring metadata storage remains bounded and up-to-date. [1] [2]

HTTP Diagnostics

  • HttpDiagnosticListener now sweeps and removes stale in-flight events (DNS, TLS, socket, HTTP) after a configurable age, preventing unbounded growth if events are missed. [1] [2] [3] [4] [5]

CI/CD Workflow Updates

  • Updated GitHub Actions in workflow files to use newer versions: docker/login-action to v4.8.0 and actions/setup-dotnet to v6 for improved reliability and security. [1] [2] [3] [4]

These changes collectively improve observability, reliability, and resource management in the application.

dependabot Bot and others added 17 commits November 10, 2026 19:35
Bumps [docker/login-action](https://github.com/docker/login-action) from 4.7.0 to 4.8.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@81c6f93...8aa5831)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 4.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5 to 6.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](actions/setup-dotnet@v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…rops

- the bounded queue can be counted, so the retention gauge reads its depth
…ancelled before the pipeline is stopped

- a .NET call that takes it returns on a stop instead of the worker being abandoned
- run-retention.ps1: settle, run N jobs or the leak workload, settle, diff holder counts and heap types
- run-stoptoken-ab.ps1: a timed-out task blocked in an HTTP call, abandoned or stopped
- docker-compose.bg-cipp.yml loads a CIPP checkout's built modules into the workers
…docker/login-action-4.8.0

chore(deps): bump docker/login-action from 4.7.0 to 4.8.0
…actions/setup-dotnet-6

chore(deps): bump actions/setup-dotnet from 5 to 6
@Zacgoose
Zacgoose requested a review from JohnDuprey November 11, 2026 23:29
- remove the _scriptTrigger/_orchestratorTrigger/_cancelTrigger response scan
- add SchedulerBridge.RunNow, which fires only timers from the scheduler config
- ExecBackendProcess starts timers through RunNow
- background command lookup never resolves HTTP entrypoints
@Zacgoose
Zacgoose merged commit c5233ca into main Nov 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants