Get DAAC credentials from MAAP when there are no Earthdata credentials - #54
Merged
Merged
Conversation
A MAAP DPS worker has no Earthdata credentials of any kind: it runs as root
with no ~/.netrc, and earthaccess's netrc and environment strategies both come
up empty. What it does have is MAAP's own auth -- $MAAP_PGT is set in the
container -- and maap.aws.earthdata_s3_credentials() exchanges that for the
DAAC's short-lived read credentials. See docs.maap-project.org,
science/NISAR/NISAR_access.html cell [2].
get_s3fs(daac=...) now tries that broker first and falls back to
earthaccess.get_s3fs_session() otherwise, so nothing changes off MAAP: without
$MAAP_PGT the new path costs one dict lookup and returns None. A DAAC with no
endpoint listed falls through the same way. Every other failure -- maap-py
missing, the broker refusing, a malformed response -- warns before falling
back, rather than quietly yielding a session from somewhere other than where
the caller thinks, which is the kind of thing that only surfaces much later as
an unexplained permission error.
find_ATL11_granules no longer hard-codes earthaccess.login(strategy='netrc').
That was the single strategy a DPS worker cannot satisfy, and it raised
LoginStrategyUnavailable before the search ever reached CMR -- for a CMR
metadata search, which needs no authentication at all. Only the granule READS
need credentials, and those go through get_s3fs(). Bare login() still picks up
whatever a local user has (environment, then netrc, then interactive), and
failure now warns instead of raising.
Verified end to end from the MAAP ADE before this was committed:
- CMR granule search with no auth: OK
- _s3fs_from_maap('NSIDC') returns a working S3FileSystem
- fs.info() and a real read of
s3://nsidc-cumulus-prod-protected/ATLAS/ATL11/007/2019/03/29/
ATL11_000103_0331_007_04.h5 -> 3979910 bytes, first 8 bytes \x89HDF...
- an unlisted DAAC returns None, so the earthaccess path is unchanged
Note that fs.ls() on the bucket ROOT is denied by NSIDC policy even with valid
credentials; object reads are what work, and what we do.
One behaviour change worth knowing: in the ADE, where $MAAP_PGT is also set,
DAAC sessions now come from MAAP rather than earthaccess. That is the path
just verified above, and it falls back with a warning if the broker declines.
The credentials are short-lived and get_s3fs() caches the session for the life
of the process. Fine for a per-tile job of minutes; a process outliving the
token's `expiration` would need to re-derive, and nothing does that yet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013MdMEXw2s6eeGtaGcaDxpF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A MAAP DPS worker has no Earthdata credentials of any kind: it runs as root with no ~/.netrc, and earthaccess's netrc and environment strategies both come up empty. What it does have is MAAP's own auth -- $MAAP_PGT is set in the container -- and maap.aws.earthdata_s3_credentials() exchanges that for the DAAC's short-lived read credentials. See docs.maap-project.org, science/NISAR/NISAR_access.html cell [2].
get_s3fs(daac=...) now tries that broker first and falls back to earthaccess.get_s3fs_session() otherwise, so nothing changes off MAAP: without $MAAP_PGT the new path costs one dict lookup and returns None. A DAAC with no endpoint listed falls through the same way. Every other failure -- maap-py missing, the broker refusing, a malformed response -- warns before falling back, rather than quietly yielding a session from somewhere other than where the caller thinks, which is the kind of thing that only surfaces much later as an unexplained permission error.
find_ATL11_granules no longer hard-codes earthaccess.login(strategy='netrc'). That was the single strategy a DPS worker cannot satisfy, and it raised LoginStrategyUnavailable before the search ever reached CMR -- for a CMR metadata search, which needs no authentication at all. Only the granule READS need credentials, and those go through get_s3fs(). Bare login() still picks up whatever a local user has (environment, then netrc, then interactive), and failure now warns instead of raising.
Verified end to end from the MAAP ADE before this was committed:
One behaviour change worth knowing: in the ADE, where $MAAP_PGT is also set, DAAC sessions now come from MAAP rather than earthaccess. That is the path just verified above, and it falls back with a warning if the broker declines.
The credentials are short-lived and get_s3fs() caches the session for the life of the process. Fine for a per-tile job of minutes; a process outliving the token's
expirationwould need to re-derive, and nothing does that yet.Claude-Session: https://claude.ai/code/session_013MdMEXw2s6eeGtaGcaDxpF