Skip to content

STAC: Fixes, Custom Provider, Item Search, Collections Search - #14

Open
rajadain wants to merge 13 commits into
deployfrom
tt/usgs-stac-collections-search
Open

rajadain wants to merge 13 commits into
deployfrom
tt/usgs-stac-collections-search

Conversation

@rajadain

@rajadain rajadain commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Serves the USGS Water Mission Area STAC catalog through pygeoapi as a spec-compliant STAC API.

The generic pygeoapi Postgres provider flattens every column into GeoJSON properties, which is fine for Features but produces invalid STAC (buried assets, doubly-nested properties, no top-level collection/bbox). This PR adds a provider that reshapes rows correctly, then fixes the STAC /search endpoint up to spec so pystac-client can actually drive it end-to-end, and finally extends the same provider to serve /collections (the collection-search extension) using the same reshape/scoping machinery.

Custom provider (pygeoapi/provider/stac_sql.py)

STACSQLProvider subclasses the existing PostgreSQLProvider and overrides only the row → document reshape and the query-filter seam, reusing the parent's engine, reflection, and paging. A mode key (items | collections) picks which STAC document type the configured table holds and which reshape runs, since a single provider instance is still bound to one table. items mode also supports scoping a resource to one collection value, so one stac_items table can back many single-collection pygeoapi resources. collections mode adds two additional filters: datetime as an interval overlap against start_datetime/end_datetime (NULL = open), and a free-text q term as an ILIKE over title/description, threaded through query().

Ultimately this Custom Provider may live in the USGS STAC repo. See stac!262. We add it here to get it onto the USGS Staging site.

Item search (pygeoapi/api/stac.py)

A series of fixes bring /stac-api/search in line with the needs of pystac-client. STAC's POST search body doesn't map 1:1 onto pygeoapi's GET query-param model, so the POST handler translates each STAC convention into the args pygeoapi understands: collections/ids become CQL IN (...) predicates against the resource's configured id/collection fields, sortby's {field, direction} objects become pygeoapi's +/-field tokens, intersects becomes a cql2-json S_INTERSECTS term, and an explicit filter (cql2-text or cql2-json) is preserved and AND-combined with the above. Filter/sort/CQL2 conformance classes are advertised to match.

Separately, STAC Items were carrying source-catalog links (e.g. ../collection.json) straight through to API responses, which broke pystac-client's navigation the moment it followed a self link; item responses now get their nav links (self/root/parent/collection) rewritten to absolute API URLs while portable links (e.g. cite-as) pass through untouched.

A new _stac_api_resources(api, mode) helper (added alongside the collections work) lets search() fan out only over items-mode resources, once collections-mode resources also exist.

Collections search (pygeoapi/api/stac.py, flask_app.py, starlette_app.py)

With the provider able to serve collections mode, this adds the API: GET /stac-api/collections (listing + collection-search: bbox, datetime, limit, sortby, filter, q), GET /stac-api/collections/{id}, GET /stac-api/collections/{id}/items, and GET /stac-api/collections/{id}/items/{itemId}. The collections listing deliberately reuses the same OGC feature-item machinery as item search (get_collection_items) rather than a new query path, then reshapes the result into a STAC Collections document, same pattern as Item search.

items/{id} closes a gap the link-rewrite work surfaced: item self links now resolve to a real 200 instead of 404, since until now nothing served a single item by collection + id. Link rewriting for collections (_rewrite_collection_links) mirrors the item version.

Routes are wired into both the Flask and Starlette apps; the deployed container serves Starlette.

rajadain and others added 13 commits September 17, 2026 10:59
The STAC API search endpoint was ignoring the collections query
parameter, returning items from all configured STAC collections
regardless of filtering. This is because the search function iterated
over every configured stac-collection resource and queried its provider
without constraining by the requested collection IDs.

Fix this by parsing the collections parameter (from GET query string or
POST JSON body) and injecting a CQL-text filter on the provider's
id_field before delegating to get_collection_items. For example,
collections=conus404-daily becomes filter="id = 'conus404-daily'" which
the SQL provider translates into a WHERE clause against the database.

The original request args are saved before filter injection and restored
afterward so that paging links reflect the user's original query
parameters rather than the internal CQL translation.
The current PostgreSQL provider puts all columns into
`properties`, which is correct for a GeoJSON Feature,
but not a STAC Item, which needs properties like bbox,
collection, stac_version, assets, and links to be top
level.

Thus, we introduce a new custom STAC PostgreSQL provider
which inherits all the essentials from the PostgreSQL
provider, while adding the custom serialization necessary
for proper STAC construction.
Previously, the STAC items were rendered without a self link.
If the source ingest provided relative links to resources, they
would be published directly via the STAC API, causing downstream
issues in pystac client:

> STACError: Relative path ../collection.json encountered without owner "self" link set.

This now converts all published links to absolute URLs, which
pystac client can use to navigate around the catalog.
Adds conversion from the STAC conventions of
[{field, direction}] in the POST body to pygeoapi's
sortby string. This allows POST based clients, like
pystac-client, to sort. Previously only GET sorting
was supported.
Previously the ids filter was ignored. Now
we translate it to a CQL `id in (...)` expression.

CQL escaping is extracted into a `_cql_in` helper.
Add filter / filter-lang to the POST query intercept
so cql2-text filters are preserved.

cql2-json object filter support will arrive in a
future commit.
These advertise filter and sort capabilities.
Notably, pystac-client defaults to cql2-json for filtering,
which is currently ignored. That will be properly implemented
in the near future.
pygeoapi will use cql2-json filtering if given in
the body. If we receive a filter object in the search
query, it replaces the full request body to activate
the cql2-json parsing path.

This now adds native support for the conventional
catalog.search(filter={}) expression of pystac-client.

Notably, this does not combine with collections / ids filters,
which are injected as cql2-text filters, thus overriding the
cql2-json body. That is a rare combination, we can revisit
this in the future if it becomes a common problem.

Also add relevant conformance classes.
Previously the collections filter was erroneously
filtering on the STAC item id field. This was leftover
from when the search endpoint was searching collections.
Now it correctly filters the collections field, as
defined in the configuration of the provider.
Translates intersects GeoJSON into a cql2-json
S_INTERSECTS filter, AND combined with any other
explicit filters.
Introduce a `mode` provider key so a single provider class
can serve both STAC Items and Collections from their
respective tables, rather than duplicating the reshape and
scoping machinery across two near-identical classes.

The single-table limit is a property of the inherited
PostgreSQLProvider (one reflected model per instance), not
of pygeoapi itself, so `mode` selects the reshape per
configured resource: `items` (default) keeps today's
behaviour byte-for-byte; `collections` is a recognised,
planned mode that fails fast until the collections-search
work builds it out.

No item behaviour changes. Config must now name
STACSQLProvider and may set `mode: items` explicitly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reshape stac_collections rows into valid STAC Collections: lift
the STAC top-level fields and JSONB extension blobs to the top
level and drop the derived query-acceleration columns (geometry,
the datetime bounds, created/updated).

Collections mode also adds the two collection-search filters the
items path lacked: datetime becomes an interval overlap against
the start_datetime/end_datetime columns (NULL bounds treated as
open), and a free-text q term becomes a case-insensitive ILIKE
over the title/description columns, threaded through query() as a
sentinel property so the parent's filter machinery is reused.

Items mode behaviour is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the STAC API Collections resource and the collection-search
extension on top of the collections-mode provider:

- GET /stac-api/collections -- listing + search (bbox, datetime,
  limit, sortby, filter, and q free-text), reusing the OGC feature
  item machinery and reshaping the result into a Collections doc.
- GET /stac-api/collections/{collectionId} -- a single Collection.
- GET /stac-api/collections/{collectionId}/items -- items in one
  collection (delegates to search, scoped to the collection).
- GET /stac-api/collections/{collectionId}/items/{itemId} -- a
  single Item, so item self links resolve (previously 404).

Navigation links are rewritten to absolute STAC API links
(_rewrite_collection_links mirrors _rewrite_item_links). The
landing page advertises the collections + collection-search
conformance classes and a data link. Routes are wired into both
the Flask and Starlette apps.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rajadain
rajadain force-pushed the tt/usgs-stac-collections-search branch from 9ea7626 to 81a318c Compare September 17, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant