Skip to content

fix(telemetry): avoid duplicate span export under auto-instrumentation wrapper - #404

Draft
jeanscherf wants to merge 1 commit into
mainfrom
fix/trace-dedup-duplicate-exporter
Draft

jeanscherf wants to merge 1 commit into
mainfrom
fix/trace-dedup-duplicate-exporter

Conversation

@jeanscherf

Copy link
Copy Markdown
Member

Disclaimer: No SAP-internal or customer-specific information is included in this PR.

Description

When an OpenTelemetry auto-instrumentation agent (for example the OTel Operator's inject-python) is active, it installs a TracerProvider that already exports spans via OTLP. auto_instrument() then adds the SDK's own Traceloop span exporter to that same provider, so every span is exported twice. The log and metric providers already avoid adding a second exporter when a provider is pre-installed; the trace path did not.

This adds _drop_wrapper_span_exporter_if_sdk_exports() (called from auto_instrument() right after the existing resource-attr merge). When the pre-installed provider carries the telemetry.auto.version marker and both the wrapper's exporter and the SDK's Traceloop exporter are present, it removes the wrapper's exporting span processor and keeps the SDK's (which also applies GenAI attribute normalization). It is a no-op when the SDK exporter is absent or no wrapper is present.

Related Issue

No linked public issue (found during internal observability work).

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Code refactoring
  • Dependency update

How to Test

  1. In an environment where an OTel auto-instrumentation agent pre-installs an exporting TracerProvider (resource marker telemetry.auto.version), call auto_instrument() and confirm the active provider ends with a single exporting span processor (the SDK's), not two.
  2. Run the new unit tests: pytest tests/core/unit/telemetry/test_auto_instrument.py -k DropWrapper.
  3. Confirm the pure-SDK path (no wrapper) is unaffected: auto_instrument() leaves the SDK exporter in place.

Checklist

  • I have read the Contributing Guidelines
  • I have verified that my changes solve the issue
  • I have added/updated automated tests to cover my changes
  • All tests pass locally
  • I have verified that my code follows the Code Guidelines
  • I have updated documentation (if applicable)
  • I have added type hints for all public APIs
  • My code does not contain sensitive information (credentials, tokens, etc.)
  • I have followed Conventional Commits for commit messages

Additional Notes

Draft, pending two decisions:

  • Direction. This consolidates span export on the SDK's exporter and removes the auto-instrumentation wrapper's duplicate exporter when both are present, so the SDK becomes the sole span transport in that configuration. That is intentional and anticipates retiring the operator auto-instrumentation layer for these services. Reviewers should weigh that the platform-managed exporter is the one removed. An alternative keeps the wrapper's exporter and drops the SDK's; the cleanest long-term shape is to move GenAI normalization into an on-end span processor so a single exporter emits normalized spans regardless of which one it is.
  • Validation. Reproduced locally (operator-style provider plus auto_instrument() goes from two exporting processors to one; pure-SDK path stays at one). Not yet validated on a deployed service.

Implementation note: the guard reads TracerProvider._active_span_processor because the OTel SDK exposes no public API to remove a span processor; this mirrors the existing _merge_resource_attrs_into_active_provider_if_wrapper_installed helper, which already mutates provider internals.

Recommend running /review-pr and /security-review on this PR before marking it ready.

…n wrapper

When an OTel auto-instrumentation agent (for example the OpenTelemetry Operator's
inject-python) is active, it installs a TracerProvider that already exports spans via OTLP.
auto_instrument() then adds the SDK's own Traceloop exporter to the same provider, so every
span is exported twice. The log and metric providers already refuse a second exporter in
this situation; the trace path did not.

Drop the wrapper's span exporter when the SDK exporter is present, consolidating on the SDK
exporter (which also applies GenAI attribute normalization). No-op when the SDK exporter is
absent or no wrapper is present, so once the auto-instrumentation layer is retired this
leaves the SDK exporter as the only one. Adds unit tests for both branches.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant