Skip to content

Read rasters, indexes and tiling schemas from s3:// as well as disk - #51

Merged
SmithB merged 1 commit into
mainfrom
s3_static_paths
Sep 4, 2026
Merged

SmithB merged 1 commit into
mainfrom
s3_static_paths

Conversation

@SmithB

@SmithB SmithB commented Sep 4, 2026

Copy link
Copy Markdown
Owner

A MAAP DPS worker has no workspace mount, so the masks, ancillary grids and ATL11 geoIndex that a tile solve reads have to come from the bucket directly. Teach the readers to take a URI rather than making every caller localize files.

Credentials are the subtle part. get_s3fs() grew daac=None, which returns a plain s3fs.S3FileSystem on the default AWS credential chain, alongside the existing earthaccess DAAC sessions. Everything here defaults to it, because these are buckets we own: an earthaccess session is scoped to a DAAC and does not grant access to them. The DAAC path is unchanged, and callers reading a granule still pass their session explicitly.

Per reader:

  • from_geotif routes through the new as_gdal_path(), since GDAL cannot open an s3:// URI but reads the same object through /vsis3/, which uses the ordinary AWS credential chain.
  • h5_open opens a remote file through s3fs and hands the object to h5py/bz2/gzip, all of which accept one in place of a name.
  • nc_open cannot do that -- netCDF4.Dataset takes a name or an in-memory buffer, never a file object -- so a remote read buffers the object and uses memory=, the same thing the compression branches already did. Fine for ancillary grids, which is what from_nc reads here.
  • tilingSchema.from_file and geoIndex.from_file take a remote path and an fs=. A remote schema's directory default then resolves to the remote directory holding its tiles, which is what makes a cloud crossover tree possible.

query_ATL11_cloud had two bugs on that path, both of which only appear once the index is remote:

  • the index existence check was os.path.isfile(), which is False for ANY URI. Every granule would have been warned-and-skipped and every tile would have come back EMPTY rather than failing -- the worst kind of failure at a fan-out of thousands of tiles. It now checks against the bucket.
  • the index and the granule need DIFFERENT credentials: the index is ours, the granule is the DAAC's. Added index_fs= rather than overloading fs=, which stays the granule's earthaccess session.

tests/test_cloud_paths.py covers all of it with a stand-in filesystem that maps s3:// URIs onto local files, so the remote branches are exercised for real with no network. 211 pass.

Claude-Session: https://claude.ai/code/session_01GTejqEYk4hmTuDC83bJiMU

A MAAP DPS worker has no workspace mount, so the masks, ancillary grids and
ATL11 geoIndex that a tile solve reads have to come from the bucket directly.
Teach the readers to take a URI rather than making every caller localize files.

Credentials are the subtle part.  get_s3fs() grew daac=None, which returns a
plain s3fs.S3FileSystem on the default AWS credential chain, alongside the
existing earthaccess DAAC sessions.  Everything here defaults to it, because
these are buckets we own: an earthaccess session is scoped to a DAAC and does
not grant access to them.  The DAAC path is unchanged, and callers reading a
granule still pass their session explicitly.

Per reader:
  - from_geotif routes through the new as_gdal_path(), since GDAL cannot open an
    s3:// URI but reads the same object through /vsis3/, which uses the ordinary
    AWS credential chain.
  - h5_open opens a remote file through s3fs and hands the object to
    h5py/bz2/gzip, all of which accept one in place of a name.
  - nc_open cannot do that -- netCDF4.Dataset takes a name or an in-memory
    buffer, never a file object -- so a remote read buffers the object and uses
    memory=, the same thing the compression branches already did.  Fine for
    ancillary grids, which is what from_nc reads here.
  - tilingSchema.from_file and geoIndex.from_file take a remote path and an fs=.
    A remote schema's directory default then resolves to the remote directory
    holding its tiles, which is what makes a cloud crossover tree possible.

query_ATL11_cloud had two bugs on that path, both of which only appear once the
index is remote:

  - the index existence check was os.path.isfile(), which is False for ANY URI.
    Every granule would have been warned-and-skipped and every tile would have
    come back EMPTY rather than failing -- the worst kind of failure at a
    fan-out of thousands of tiles.  It now checks against the bucket.
  - the index and the granule need DIFFERENT credentials: the index is ours, the
    granule is the DAAC's.  Added index_fs= rather than overloading fs=, which
    stays the granule's earthaccess session.

tests/test_cloud_paths.py covers all of it with a stand-in filesystem that maps
s3:// URIs onto local files, so the remote branches are exercised for real with
no network. 211 pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GTejqEYk4hmTuDC83bJiMU
@SmithB
SmithB merged commit 1adc175 into main Sep 4, 2026
0 of 2 checks passed
@SmithB
SmithB deleted the s3_static_paths branch September 4, 2026 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant