Repository navigation
feat(dump): download a resource's data dictionary and metadata - #32
Merged
Merged
Conversation
Add two file downloads next to the table dump, each `?format=csv|json`:
- `dump/{resource_id}/dictionary` — the resource's Frictionless schema
(as `datastore_info` returns it). CSV is one row per column, one column
per field property; nested values are written as JSON in the cell.
- `dump/{resource_id}/metadata` — the resource's and its dataset's
metadata (datopian/neso-nextgen-shared#42). The dataset is read with a
new `CKANClient.package_show`; data group is the organization title,
"last updated" matches the portal (`metadata_modified`, else
`last_modified`), and the record count comes from the engine. CSV is a
header row plus one value row. Requires AUTH_TYPE=ckan.
`dump_url` moves to core/helper so the metadata's Download URL matches
the `url` datastore_create writes. Also drops a stray debug print of
every CKAN response status.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements datopian/neso-nextgen-shared#42 (CH15: download the data dictionary, plus the dataset/resource metadata requested in the comments).
New endpoints
Both need
readon the resource, take?format=csv|json(defaultcsv) and return the file directly (200,Content-Dispositionnamed like the table dump, e.g.<name>_data_dictionary.csv/<name>_metadata.json). They're small, so they're built in memory. There's no export or redirect.GET /datastore/api/v2/dump/{resource_id}/dictionarydatastore_inforeturns it.name, type, title, descriptionfirst and then every other property. Nested values (e.g.constraints) are compact JSON in their cell.GET /datastore/api/v2/dump/{resource_id}/metadata(AUTH_TYPE=ckanonly)The fields from the issue. The dataset is read via a new
CKANClient.package_showrather than the cacheddatastore_authorizepayload.organization.titlemetadata_modified, elselast_modified(same as the portal shows)dump/{resource_id}URLtemporal_coveragesplit on/CSV is a header row plus one value row, with tags comma-joined. JSON is one object with the same labels as keys.
Other changes
_dump_urlmoved fromservices/write.pytocore/helper.dump_url, so the Download URL matches theurlthatdatastore_createwrites to CKAN.print(response.status_code)inCKANClient._unwrap.datastore_dump_dictionary/datastore_dump_metadata.API.md/CLAUDE.mdupdated.Testing
pytest: 568 passed (26 new intests/test_datastore_dump_documents.py).