Skip to content

Harden CKAN catalog mirroring for full source catalogs - #14

Merged
minhajuddin2510 merged 3 commits into
mainfrom
codex/public-ckan-mirroring
Sep 22, 2026
Merged

minhajuddin2510 merged 3 commits into
mainfrom
codex/public-ckan-mirroring

Conversation

@minhajuddin2510

Copy link
Copy Markdown
Collaborator

Problem

The first production mirror exposed three cases that the local fixture did not cover: WPRDC caps organization listings at 25 rows, its catalog has a publisher named wprdc that collides with the synthetic portal root, and Cloudflare can return a 502 after CKAN has already committed a resource.

Change

  • paginate organization and group listings until the source is exhausted
  • use wprdc-source when the source already owns the bare portal slug, while safely migrating an earlier synthetic root
  • supply a neutral description only when the destination schema rejects an empty source description
  • reconcile preserved UUIDs after failed create responses and retry transient failures

Validation

  • Ruff passes
  • 14 focused unit tests pass
  • production dry run now discovers 31 organizations, 371 datasets, 2,545 resources, and 110 qsv-enriched resources
  • the production mirror resumed successfully across previously completed records

@minhajuddin2510
minhajuddin2510 merged commit bb56715 into main Sep 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant