Repository navigation
Conversation
The Rust and Python SDKs (dataplatform-rust-sdk #177) gain the binary read, buffered and streamed, and a binary ingest straight from Arrow: insert_dataframe_binary in Python, insert_table_binary in Rust. Their binary ingest now sends compression 2 by default, with "payload" for compression 1, and the default zstd level is 3 rather than 9. Every Python and Rust snippet added here ran against a local platform. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jgjesdal <jostein@intellistream.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the Rust and Python half of the binary datapoint work, IntelliStream-DataHub/dataplatform-rust-sdk#177. Stacked on #114, which documents the platform and Java side of the same read; merge them together, once both the platform PR and SDK #177 are on
main.Closes #115.
docs/reference/timeseries.md2by default, so a platform older than 1.1 refuses their frames. Python and Rust can send compression1withcompression="payload"/FrameCompression::Payload.insert_dataframe_binary(polars, pandas, pyarrow, without copying), with the rules for the timestamp column, gaps, ordering and refusals.insert_table_binaryfor Arrow record batches.BinaryIngestOptionshas four knobs now,compressionincluded.retrieve_datapoints_binarywithto_polars(wide=False/True), and theretrieve_datapoints_binary_streamreader.retrieve_datapoints_binaryandstream_datapoints_binary.Other pages
docs/guides/ingest-timeseries.mdx: the Python tab leads withinsert_dataframe_binary; the zstd default comments are fixed.docs/guides/query-and-aggregate.mdx: the large-read paragraph names all three SDKs.docs/reference/binary-datapoints.md: which SDK sends which compression, and which calls read and write frames.docs/reference/limits.md: the binary options and their defaults.Verification
🤖 Generated with Claude Code