Skip to content

fix(process manager): apply limit and offset in get_jobs - #2427

Open
C1-BA-B1-F3 wants to merge 1 commit into
geopython:masterfrom
C1-BA-B1-F3:fix/jobs-limit-offset
Open

C1-BA-B1-F3 wants to merge 1 commit into
geopython:masterfrom
C1-BA-B1-F3:fix/jobs-limit-offset

Conversation

@C1-BA-B1-F3

Copy link
Copy Markdown
Contributor

Summary

The PostgreSQL and MongoDB process managers ignored the limit and offset parameters in get_jobs, returning all jobs regardless of the query string. The TinyDB manager already applied both correctly.

This PR adds offset then limit to both backends, matching the TinyDB behaviour.

Changes

  • pygeoapi/process/manager/postgresql.py: add .offset() / .limit() to the SQLAlchemy query
  • pygeoapi/process/manager/mongodb_.py: add .skip() / .limit() to the PyMongo cursor

Test plan

  • pytest tests/manager/test_manager.py — 4/4 pass
  • pytest tests/manager/test_tinydb_manager_for_parallel_requests.py — 1/1 pass
  • PostgreSQL integration tests require a live database and were not run locally; the change is confined to the query builder and mirrors the existing TinyDB pattern.

Fixes #2426

PostgreSQL and MongoDB managers ignored the limit and offset
parameters, returning all jobs regardless of the query string.

Apply offset before limit in both backends, matching the existing
TinyDB behaviour.

Fixes geopython#2426
@C1-BA-B1-F3

C1-BA-B1-F3 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Correcting the CI evidence in my earlier comment: the Build failure has a single root cause, it is not caused by this PR, and my earlier pointer to the 2026-09-29 templating run was wrong (that run failed only on test_postgresql_provider with ModuleNotFoundError: No module named 'psycopg', not on these two tests).

Root cause

The usgs-prism collection fails to load:

WARNING pygeoapi.provider.xarray_:xarray_.py:114
        open_group() got an unexpected keyword argument 'zarr_format'
WARNING pygeoapi.api:__init__.py:956
        Error generating collection usgs-prism: open_group() got an unexpected
        keyword argument 'zarr_format'

That single failure produces both red tests: describe_collections sees 9 collections instead of 10, and every EDR query against usgs-prism returns 500 (ProviderNoDataError -> test_get_collection_edr_query).

Why it is environmental, not this PR

  • The last green Build runs — master (2026-09-22) and the psycopg branch (2026-09-29) — installed xarray 2026.7.0.
  • The runs from 2026-10-03 install xarray 2026.9.0. zarr is unchanged at 2.18.7 (pinned <3).
  • requirements-provider.txt leaves xarray unpinned, and 2026.9.0 changes the Zarr open path, which breaks pygeoapi/provider/xarray_.py for the usgs-prism Zarr store.

This PR only touches pygeoapi/process/manager/postgresql.py` and `pygeoapi/process/manager/mongodb_.py; it does not touch the EDR/xarray provider or the collection config. Happy to rebase once the environment (or the xarray pin) is fixed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

the query parameter limit is not working for jobs

1 participant