Skip to content

fix(telemetry): request the cloud-platform scope for GCP OTel exporters - #7025

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7024-cloud-telemetry-invalid-scope
Open

fix(telemetry): request the cloud-platform scope for GCP OTel exporters#7025
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7024-cloud-telemetry-invalid-scope

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Closes: #7024

Problem

With get_fast_api_app(..., otel_to_cloud=True) and Application Default
Credentials supplied as a service-account key file, every Cloud Trace
and Cloud Monitoring export fails at the token grant, before any HTTP
request is made:

google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', ...)
  ... opentelemetry/exporter/otlp/proto/http/{trace,metric}_exporter/__init__.py
  ... google/auth/transport/requests.py  request
  ... google/oauth2/service_account.py   _perform_refresh_token
  ... google/oauth2/_client.py           jwt_grant

The running application is unaffected — only the exporters fail — so the
symptom is a continuous stream of tracebacks in the container log with no
telemetry ever arriving.

Root cause

google.auth.default() is called with no scopes at the three places that
resolve ADC for the GCP OTel exporters:

  • src/google/adk/cli/api_server.py_setup_gcp_telemetry, the path that
    fires from get_fast_api_app.
  • src/google/adk/telemetry/google_cloud.py — the google_auth is None
    fallback in get_gcp_exporters and in _get_gcp_otlp_metric_exporter.

GCE/GKE/Cloud Run metadata-server credentials tolerate an unscoped request,
so this is invisible in those environments. A service-account key file has
requires_scopes = True and no scopes, so its token refresh is rejected.

Fix

Request the https://www.googleapis.com/auth/cloud-platform scope
explicitly at all three call sites, via google.auth.default(scopes=[...])
(which routes through google.auth.credentials.with_scopes_if_required
internally, so credentials that don't need a scope are left untouched).

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Added assertions to test_get_gcp_exporters and
test_get_gcp_otlp_metric_exporter_uses_default_credentials in
tests/unittests/telemetry/test_google_cloud.py, and a new
test_setup_gcp_telemetry_requests_cloud_platform_scope in
tests/unittests/cli/test_fast_api.py, each asserting
google.auth.default is called with scopes=[CLOUD_PLATFORM_SCOPE].

Confirmed the new/updated assertions fail without the fix
(git checkout HEAD~1 -- src/google/adk/cli/api_server.py src/google/adk/telemetry/google_cloud.py):

FAILED tests/unittests/telemetry/test_google_cloud.py::test_get_gcp_exporters[...] - AttributeError: module 'google.adk.telemetry.google_cloud' has no attribute 'CLOUD_PLATFORM_SCOPE'
FAILED tests/unittests/telemetry/test_google_cloud.py::test_get_gcp_otlp_metric_exporter_uses_default_credentials - AttributeError: ...
FAILED tests/unittests/cli/test_fast_api.py::test_setup_gcp_telemetry_requests_cloud_platform_scope - ImportError: cannot import name 'CLOUD_PLATFORM_SCOPE' ...
10 failed, 1 passed, 168 deselected

And passes with the fix restored:

$ python3 -m pytest tests/unittests/telemetry/test_google_cloud.py tests/unittests/cli/test_fast_api.py -q
173 passed, 5 skipped, 1 xfailed, 16 warnings in 23.33s

Full unit suite also passes:

$ python3 -m pytest tests/unittests -q -n auto
13981 passed, 82 skipped, 27 xfailed, 2 xpassed, 2141 warnings, 24 subtests passed in 154.32s

Formatting/lint checked with the repo's configured tools:

$ python3 -m pyink --check src/google/adk/cli/api_server.py src/google/adk/telemetry/google_cloud.py tests/unittests/cli/test_fast_api.py tests/unittests/telemetry/test_google_cloud.py
All done! 4 files would be left unchanged.

$ python3 -m isort --check-only src/google/adk/cli/api_server.py src/google/adk/telemetry/google_cloud.py tests/unittests/cli/test_fast_api.py tests/unittests/telemetry/test_google_cloud.py
(no changes)

$ pre-commit run --files src/google/adk/cli/api_server.py src/google/adk/telemetry/google_cloud.py tests/unittests/cli/test_fast_api.py tests/unittests/telemetry/test_google_cloud.py
... all hooks Passed

Manual End-to-End (E2E) Tests:

Not run — reproducing the failure requires an off-GCP host with
GOOGLE_APPLICATION_CREDENTIALS pointed at a service-account key file, which
isn't available in this environment. The unit tests above assert the exact
google.auth.default(scopes=[...]) call the issue's reproduction script
shows fixes the RefreshError.

Additional context

This PR intentionally addresses only the invalid_scope failure described
in the issue. The issue also separately mentions a follow-on
INVALID_ARGUMENT: prometheus_target resource type must have an instance specified error from get_gcp_resource() on non-GCP hosts, which the
reporter offered to file separately — left out of scope here.


Developed with AI assistance (Claude Code), reviewed and tested by a human before submission.

google.auth.default() is called with no scopes to build the Cloud Trace
/ Cloud Monitoring / Cloud Logging exporters. GCE/GKE/Cloud Run metadata
credentials tolerate that, but a service-account key file has
requires_scopes=True and no scopes, so its token refresh fails with
invalid_scope before any export request is made -- only the exporters
are affected, not the running application. Request the cloud-platform
scope explicitly wherever ADC is resolved for these exporters.

Fixes google#7024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cloud telemetry export fails with invalid_scope when ADC is a service-account key file

2 participants