Skip to content

fix(metrics): shutdown periodic readers in tests and handle unmapped aggregations (#5157) - #5622

Open
dlowzzxx wants to merge 3 commits into
open-telemetry:mainfrom
dlowzzxx:fix/metrics-test-thread-leak-collector-fallback-5157
Open

fix(metrics): shutdown periodic readers in tests and handle unmapped aggregations (#5157)#5622
dlowzzxx wants to merge 3 commits into
open-telemetry:mainfrom
dlowzzxx:fix/metrics-test-thread-leak-collector-fallback-5157

Conversation

@dlowzzxx

@dlowzzxx dlowzzxx commented Sep 4, 2026

Copy link
Copy Markdown

Description

Fixes #5157

Root Cause

  1. In opentelemetry-sdk/src/opentelemetry/sdk/metrics/_internal/metric_reader_storage.py, MetricReaderStorage.collect() iterates through view instrument matches and assigns data depending on the aggregation instance. When an unmapped or mock aggregation was encountered, no branch was taken, leaving data unassigned and resulting in UnboundLocalError: cannot access local variable 'data' where it is not associated with a value when constructing Metric(...).
  2. In opentelemetry-sdk/tests/metrics/test_periodic_exporting_metric_reader.py, test_exporter_temporality_preference and test_exporter_aggregation_preference initialized a PeriodicExportingMetricReader which started a background daemon thread (_ticker), but never shut down pmr. Similarly, in test_metric_reader_metrics, mp.shutdown() was not guaranteed to execute in a finally block if an assertion failed. These leaking daemon threads continued ticking during subsequent tests (such as test_boundary_statistics and test_creates_view_instrument_matches) and triggered unhandled thread exceptions in the background.

Changes

  1. MetricReaderStorage: Added an else: fallback in the aggregation isinstance chain that logs a warning (_logger.warning(...)) and skips the match (continue), preventing unassigned data fallthrough. Removed # pylint: disable=possibly-used-before-assignment.
  2. Periodic Exporting Metric Reader Tests: Wrapped reader inspection in try...finally: pmr.shutdown() for test_exporter_temporality_preference and test_exporter_aggregation_preference, and ensured mp.shutdown() (and pmr.shutdown() if not yet stopped) runs inside a finally block in test_metric_reader_metrics.
  3. New Test: Added test_collect_skips_unsupported_aggregation to test_metric_reader_storage.py to assert that unsupported aggregations log a warning and are cleanly skipped without raising UnboundLocalError.
  4. Changelog Fragment: Added .changelog/5622.fixed (matching towncrier and CI PR number requirements).

Testing

  • uv run pytest opentelemetry-sdk/tests/metrics/test_metric_reader_storage.py passed (18 passed).
  • uv run pytest opentelemetry-sdk/tests/metrics/test_periodic_exporting_metric_reader.py passed (13 passed, 1 skipped for fork).
  • Full metrics test suite uv run pytest opentelemetry-sdk/tests/metrics/ passed (322 passed, 0 failures).
  • uv run towncrier check --compare-with origin/main passed cleanly.
  • uv run ruff check and uv run ruff format --check passed with zero violations.

@dlowzzxx
dlowzzxx requested a review from a team as a code owner September 4, 2026 16:30
@linux-foundation-easycla

linux-foundation-easycla Bot commented Sep 4, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Sep 4, 2026

Copy link
Copy Markdown

Pull request dashboard status

Waiting on reviewers · refreshed 2026-09-06 15:21 UTC

Review the latest changes.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

@ocelotl

ocelotl commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

please sign the CLA

@ocelotl

ocelotl commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

how does this PR fix #5157?

@dlowzzxx

dlowzzxx commented Sep 4, 2026

Copy link
Copy Markdown
Author

@ocelotl Thank you for reviewing!

Regarding how this PR addresses #5157:

The traceback captured in #5157 shows two symptoms occurring together:

  1. \AssertionError: 2 != 1\ in \TestMetricReaderStorage.test_creates_view_instrument_matches\
  2. \PytestUnhandledThreadExceptionWarning: Exception in thread OtelPeriodicExportingMetricReader: UnboundLocalError: cannot access local variable 'data' where it is not associated with a value\ in \metric_reader_storage.py:219.

These two failures share a single root cause:

  • Tests in \ est_periodic_exporting_metric_reader.py\ spawned \PeriodicExportingMetricReader\ instances whose daemon background ticker threads (_ticker) were not explicitly joined/shut down in test teardown.
  • While \TestMetricReaderStorage\ was executing, a leaked background ticker thread concurrently invoked \collect()\ -> \storage.consume_measurement(). This triggered _ViewInstrumentMatch\ a second time, causing \AssertionError: 2 != 1. Furthermore, because the mock aggregations in that test did not produce a matching aggregation case in \MetricReaderStorage.collect(), the concurrent call referenced an unassigned \data\ variable, raising \UnboundLocalError.

This PR fixes both:

  1. Adds explicit
    eader.shutdown()\ in \ inally\ blocks across \ est_periodic_exporting_metric_reader.py, preventing the background ticker thread leak.
  2. Adds defensive fallback in \MetricReaderStorage.collect()\ so unmapped aggregations do not crash with \UnboundLocalError.

@dlowzzxx

dlowzzxx commented Sep 6, 2026

Copy link
Copy Markdown
Author

/dashboard route:reviewers

@opentelemetry-pr-dashboard

Copy link
Copy Markdown

@dlowzzxx, this pull request was routed to reviewers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Failing test on windows / 3.14t

2 participants