Read rasters, indexes and tiling schemas from s3:// as well as disk - #51
Merged
Merged
Conversation
A MAAP DPS worker has no workspace mount, so the masks, ancillary grids and
ATL11 geoIndex that a tile solve reads have to come from the bucket directly.
Teach the readers to take a URI rather than making every caller localize files.
Credentials are the subtle part. get_s3fs() grew daac=None, which returns a
plain s3fs.S3FileSystem on the default AWS credential chain, alongside the
existing earthaccess DAAC sessions. Everything here defaults to it, because
these are buckets we own: an earthaccess session is scoped to a DAAC and does
not grant access to them. The DAAC path is unchanged, and callers reading a
granule still pass their session explicitly.
Per reader:
- from_geotif routes through the new as_gdal_path(), since GDAL cannot open an
s3:// URI but reads the same object through /vsis3/, which uses the ordinary
AWS credential chain.
- h5_open opens a remote file through s3fs and hands the object to
h5py/bz2/gzip, all of which accept one in place of a name.
- nc_open cannot do that -- netCDF4.Dataset takes a name or an in-memory
buffer, never a file object -- so a remote read buffers the object and uses
memory=, the same thing the compression branches already did. Fine for
ancillary grids, which is what from_nc reads here.
- tilingSchema.from_file and geoIndex.from_file take a remote path and an fs=.
A remote schema's directory default then resolves to the remote directory
holding its tiles, which is what makes a cloud crossover tree possible.
query_ATL11_cloud had two bugs on that path, both of which only appear once the
index is remote:
- the index existence check was os.path.isfile(), which is False for ANY URI.
Every granule would have been warned-and-skipped and every tile would have
come back EMPTY rather than failing -- the worst kind of failure at a
fan-out of thousands of tiles. It now checks against the bucket.
- the index and the granule need DIFFERENT credentials: the index is ours, the
granule is the DAAC's. Added index_fs= rather than overloading fs=, which
stays the granule's earthaccess session.
tests/test_cloud_paths.py covers all of it with a stand-in filesystem that maps
s3:// URIs onto local files, so the remote branches are exercised for real with
no network. 211 pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTejqEYk4hmTuDC83bJiMU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A MAAP DPS worker has no workspace mount, so the masks, ancillary grids and ATL11 geoIndex that a tile solve reads have to come from the bucket directly. Teach the readers to take a URI rather than making every caller localize files.
Credentials are the subtle part. get_s3fs() grew daac=None, which returns a plain s3fs.S3FileSystem on the default AWS credential chain, alongside the existing earthaccess DAAC sessions. Everything here defaults to it, because these are buckets we own: an earthaccess session is scoped to a DAAC and does not grant access to them. The DAAC path is unchanged, and callers reading a granule still pass their session explicitly.
Per reader:
query_ATL11_cloud had two bugs on that path, both of which only appear once the index is remote:
tests/test_cloud_paths.py covers all of it with a stand-in filesystem that maps s3:// URIs onto local files, so the remote branches are exercised for real with no network. 211 pass.
Claude-Session: https://claude.ai/code/session_01GTejqEYk4hmTuDC83bJiMU