Repository navigation
Conversation
retrieve_datapoints_binary posts the JSON retrieve body to POST /timeseries/data/list/binary and parses the frames that come back into one columnar BinaryDatapoints per series. A body without the DHDE end marker was cut short and is a 502, never fewer points. Aggregate frames (codec 2) carry avg, min, max and sum; decimals stay exact. The envelopes are walked first and the frames then decompressed and decoded on scoped threads: zstd was 90% of a single-threaded parse. For 100M float rows the parse went from 1.46s to 0.60s and the SDK's share of an end-to-end read from 1.57s to 0.90s. In Python, the sync client's retrieve_datapoints_binary returns objects implementing the Arrow PyCapsule stream protocol, so polars, pyarrow and pandas 3 take them without copying. The package adds no runtime dependency. The offline tests read fixtures written by the platform's own frame writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: jgjesdal <jostein@intellistream.ai>
This was referenced Oct 12, 2026
Contributor
Author
|
Included in #177, which now targets main and carries this PR's commits unchanged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TimeSeriesService::retrieve_datapoints_binaryposts the ordinary retrieve body toPOST /timeseries/data/list/binaryand parses the zstd-compressed Arrow frames that come back into one columnarBinaryDatapointsper series. In Python, the sync client'sretrieve_datapoints_binaryreturns objects that implement the Arrow PyCapsule stream protocol, sopolars.DataFrame(s),pyarrow.table(s)andpandas.DataFrame.from_arrow(s)take them without copying. The package adds no runtime dependency;arrow-array'sffifeature is enabled in the bindings crate only.DHDEend marker was cut short and is a 502, never fewer points; bytes after the marker are refused.avg/min/max/sum;numericanddecimal32stay exact (unscaledi128+ scale).Depends on the platform
The endpoint is the platform's
feat/binary-datapoint-read(through524ca6aa6). Don't merge before that lands: against an api without it the call is a 404. The live tests skip (Python) or are#[ignore](Rust) for that reason.Differences from the JSON read
cursor(400); read on by movingstartpast the last timestamp.limitreturns its whole window; alimitis capped at 100 000.Tests
DatapointFrameWriter.wrap(src/timeseries/fixtures/, generator committed beside them), cut-short bodies, canon violations, frame order under parallel decode.test_datapoints_binary_read(150k floats across frames, 25k texts, hourly aggregates against the JSON read, refusals) and the Python live tests all pass.Docs
Affects datahub-sdk-docs
docs/guides/query-and-aggregate.mdx(a binary read section for Rust and Python) and the Python reference built from the.pyistub.🤖 Generated with Claude Code