Skip to content

Get DAAC credentials from MAAP when there are no Earthdata credentials - #54

Merged
SmithB merged 1 commit into
mainfrom
maap_s3_credentials
Sep 8, 2026
Merged

SmithB merged 1 commit into
mainfrom
maap_s3_credentials

Conversation

@SmithB

@SmithB SmithB commented Sep 8, 2026

Copy link
Copy Markdown
Owner

A MAAP DPS worker has no Earthdata credentials of any kind: it runs as root with no ~/.netrc, and earthaccess's netrc and environment strategies both come up empty. What it does have is MAAP's own auth -- $MAAP_PGT is set in the container -- and maap.aws.earthdata_s3_credentials() exchanges that for the DAAC's short-lived read credentials. See docs.maap-project.org, science/NISAR/NISAR_access.html cell [2].

get_s3fs(daac=...) now tries that broker first and falls back to earthaccess.get_s3fs_session() otherwise, so nothing changes off MAAP: without $MAAP_PGT the new path costs one dict lookup and returns None. A DAAC with no endpoint listed falls through the same way. Every other failure -- maap-py missing, the broker refusing, a malformed response -- warns before falling back, rather than quietly yielding a session from somewhere other than where the caller thinks, which is the kind of thing that only surfaces much later as an unexplained permission error.

find_ATL11_granules no longer hard-codes earthaccess.login(strategy='netrc'). That was the single strategy a DPS worker cannot satisfy, and it raised LoginStrategyUnavailable before the search ever reached CMR -- for a CMR metadata search, which needs no authentication at all. Only the granule READS need credentials, and those go through get_s3fs(). Bare login() still picks up whatever a local user has (environment, then netrc, then interactive), and failure now warns instead of raising.

Verified end to end from the MAAP ADE before this was committed:

  • CMR granule search with no auth: OK
  • _s3fs_from_maap('NSIDC') returns a working S3FileSystem
  • fs.info() and a real read of s3://nsidc-cumulus-prod-protected/ATLAS/ATL11/007/2019/03/29/ ATL11_000103_0331_007_04.h5 -> 3979910 bytes, first 8 bytes \x89HDF...
  • an unlisted DAAC returns None, so the earthaccess path is unchanged Note that fs.ls() on the bucket ROOT is denied by NSIDC policy even with valid credentials; object reads are what work, and what we do.

One behaviour change worth knowing: in the ADE, where $MAAP_PGT is also set, DAAC sessions now come from MAAP rather than earthaccess. That is the path just verified above, and it falls back with a warning if the broker declines.

The credentials are short-lived and get_s3fs() caches the session for the life of the process. Fine for a per-tile job of minutes; a process outliving the token's expiration would need to re-derive, and nothing does that yet.

Claude-Session: https://claude.ai/code/session_013MdMEXw2s6eeGtaGcaDxpF

A MAAP DPS worker has no Earthdata credentials of any kind: it runs as root
with no ~/.netrc, and earthaccess's netrc and environment strategies both come
up empty.  What it does have is MAAP's own auth -- $MAAP_PGT is set in the
container -- and maap.aws.earthdata_s3_credentials() exchanges that for the
DAAC's short-lived read credentials.  See docs.maap-project.org,
science/NISAR/NISAR_access.html cell [2].

get_s3fs(daac=...) now tries that broker first and falls back to
earthaccess.get_s3fs_session() otherwise, so nothing changes off MAAP: without
$MAAP_PGT the new path costs one dict lookup and returns None.  A DAAC with no
endpoint listed falls through the same way.  Every other failure -- maap-py
missing, the broker refusing, a malformed response -- warns before falling
back, rather than quietly yielding a session from somewhere other than where
the caller thinks, which is the kind of thing that only surfaces much later as
an unexplained permission error.

find_ATL11_granules no longer hard-codes earthaccess.login(strategy='netrc').
That was the single strategy a DPS worker cannot satisfy, and it raised
LoginStrategyUnavailable before the search ever reached CMR -- for a CMR
metadata search, which needs no authentication at all.  Only the granule READS
need credentials, and those go through get_s3fs().  Bare login() still picks up
whatever a local user has (environment, then netrc, then interactive), and
failure now warns instead of raising.

Verified end to end from the MAAP ADE before this was committed:
  - CMR granule search with no auth: OK
  - _s3fs_from_maap('NSIDC') returns a working S3FileSystem
  - fs.info() and a real read of
    s3://nsidc-cumulus-prod-protected/ATLAS/ATL11/007/2019/03/29/
    ATL11_000103_0331_007_04.h5 -> 3979910 bytes, first 8 bytes \x89HDF...
  - an unlisted DAAC returns None, so the earthaccess path is unchanged
Note that fs.ls() on the bucket ROOT is denied by NSIDC policy even with valid
credentials; object reads are what work, and what we do.

One behaviour change worth knowing: in the ADE, where $MAAP_PGT is also set,
DAAC sessions now come from MAAP rather than earthaccess.  That is the path
just verified above, and it falls back with a warning if the broker declines.

The credentials are short-lived and get_s3fs() caches the session for the life
of the process.  Fine for a per-tile job of minutes; a process outliving the
token's `expiration` would need to re-derive, and nothing does that yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MdMEXw2s6eeGtaGcaDxpF
@SmithB
SmithB merged commit 8850b0c into main Sep 8, 2026
2 checks passed
@SmithB
SmithB deleted the maap_s3_credentials branch September 8, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant