Skip to content

docs: binary read, dataframe ingest and compression 2 in Python and Rust - #116

Open
JosteinGj wants to merge 1 commit into
docs/binary-datapoint-readfrom
docs/sdk-binary-read-and-dataframe
Open

JosteinGj wants to merge 1 commit into
docs/binary-datapoint-readfrom
docs/sdk-binary-read-and-dataframe

Conversation

@JosteinGj

Copy link
Copy Markdown
Contributor

Summary

Documents the Rust and Python half of the binary datapoint work, IntelliStream-DataHub/dataplatform-rust-sdk#177. Stacked on #114, which documents the platform and Java side of the same read; merge them together, once both the platform PR and SDK #177 are on main.

Closes #115.

docs/reference/timeseries.md

  • Binary ingest:
    • All three SDKs now send compression 2 by default, so a platform older than 1.1 refuses their frames. Python and Rust can send compression 1 with compression="payload" / FrameCompression::Payload.
    • New: Python's insert_dataframe_binary (polars, pandas, pyarrow, without copying), with the rules for the timestamp column, gaps, ordering and refusals.
    • New: Rust's insert_table_binary for Arrow record batches.
    • Python and Rust default to zstd level 3; Java stays at 9.
    • Rust's BinaryIngestOptions has four knobs now, compression included.
  • Binary read:
    • The JSON/binary comparison table covers all three SDKs and gains two rows: a series with no points is left out, and an unknown series is a 400 for the whole request.
    • The section now has Python and Rust tabs:
      • Python: retrieve_datapoints_binary with to_polars(wide=False/True), and the retrieve_datapoints_binary_stream reader.
      • Rust: retrieve_datapoints_binary and stream_datapoints_binary.
    • Both tabs explain what a response that breaks off means for each form.
  • What each client covers: the binary ingest and binary read rows list the new calls. A new row says only Java writes read frames back as they are.

Other pages

  • docs/guides/ingest-timeseries.mdx: the Python tab leads with insert_dataframe_binary; the zstd default comments are fixed.
  • docs/guides/query-and-aggregate.mdx: the large-read paragraph names all three SDKs.
  • docs/reference/binary-datapoints.md: which SDK sends which compression, and which calls read and write frames.
  • docs/reference/limits.md: the binary options and their defaults.

Verification

  • Every new Python and Rust snippet ran against a local platform. The Rust ones were compiled as an example against the SDK branch.
  • The site builds clean on Node 22.

🤖 Generated with Claude Code

The Rust and Python SDKs (dataplatform-rust-sdk #177) gain the binary
read, buffered and streamed, and a binary ingest straight from Arrow:
insert_dataframe_binary in Python, insert_table_binary in Rust. Their
binary ingest now sends compression 2 by default, with "payload" for
compression 1, and the default zstd level is 3 rather than 9.

Every Python and Rust snippet added here ran against a local platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant