fix(api): name the multipart field after the detected format in append_datasource - #36
Open
RaphaelFakhri wants to merge 1 commit into
Conversation
…d_datasource append_datasource always sent local files under the csv form field. The API requires the field name to be ndjson or parquet for those formats, so NDJSON and Parquet uploads failed. Use the format detected from the file extension and keep csv as the fallback. Closes tinybirdco#23
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
append_datasourcenow names the multipart form field after the detected file format (csv,ndjsonorparquet). Fixes #23.Problem
When you pass a local
file,append_datasourcealways sent the file under the form fieldcsv. The Tinybird API requires the field name to bendjsonorparquetfor those formats, so uploads of.ndjson,.jsonland.parquetfiles failed withNDJSON/Parquet multipart requests require name field set to 'ndjson'/'parquet'.Fix
The multipart field name is
detected_format or "csv". The format detection already existed and already set theformatquery parameter. Files with an unrecognized extension keep thecsvfield name.Testing
A new parametrized test in
tests/test_api_parity.pyuploads.csv,.ndjson,.jsonl,.parquetand.datfiles and checks the field name in the multipart body. The.ndjson,.jsonland.parquetcases fail on the previous code and pass now.make lint,make typecheckandmake testpass.Checklist
lint,typecheck,test) pass locallyCHANGELOG.mdwas updated