Skip to content

Read datapoints through the binary endpoint - #175

Closed
JosteinGj wants to merge 1 commit into
mainfrom
feat/binary-datapoint-read
Closed

JosteinGj wants to merge 1 commit into
mainfrom
feat/binary-datapoint-read

Conversation

@JosteinGj

Copy link
Copy Markdown
Contributor

What

TimeSeriesService::retrieve_datapoints_binary posts the ordinary retrieve body to POST /timeseries/data/list/binary and parses the zstd-compressed Arrow frames that come back into one columnar BinaryDatapoints per series. In Python, the sync client's retrieve_datapoints_binary returns objects that implement the Arrow PyCapsule stream protocol, so polars.DataFrame(s), pyarrow.table(s) and pandas.DataFrame.from_arrow(s) take them without copying. The package adds no runtime dependency; arrow-array's ffi feature is enabled in the bindings crate only.

  • A body without the DHDE end marker was cut short and is a 502, never fewer points; bytes after the marker are refused.
  • Aggregate frames (codec 2) come back as avg/min/max/sum; numeric and decimal32 stay exact (unscaled i128 + scale).
  • Frames are decompressed and decoded on several threads: zstd was 90% of a single-threaded parse. For 100M float rows the parse went from 1.46 s to 0.60 s.

Depends on the platform

The endpoint is the platform's feat/binary-datapoint-read (through 524ca6aa6). Don't merge before that lands: against an api without it the call is a 404. The live tests skip (Python) or are #[ignore] (Rust) for that reason.

Differences from the JSON read

  • No cursor (400); read on by moving start past the last timestamp.
  • An item without a limit returns its whole window; a limit is capped at 100 000.
  • A series with no points is absent, not empty; an unknown or unreadable series is a 400 for the whole request.

Tests

  • Offline (Rust): response bodies written by the platform's own DatapointFrameWriter.wrap (src/timeseries/fixtures/, generator committed beside them), cut-short bodies, canon violations, frame order under parallel decode.
  • Offline (Python): a stand-in server; polars and pandas conversions, exact decimals, cut-short, refusals.
  • Live, against the platform branch: test_datapoints_binary_read (150k floats across frames, 25k texts, hourly aggregates against the JSON read, refusals) and the Python live tests all pass.

Docs

Affects datahub-sdk-docs docs/guides/query-and-aggregate.mdx (a binary read section for Rust and Python) and the Python reference built from the .pyi stub.

🤖 Generated with Claude Code

retrieve_datapoints_binary posts the JSON retrieve body to
POST /timeseries/data/list/binary and parses the frames that come back
into one columnar BinaryDatapoints per series. A body without the DHDE
end marker was cut short and is a 502, never fewer points. Aggregate
frames (codec 2) carry avg, min, max and sum; decimals stay exact.

The envelopes are walked first and the frames then decompressed and
decoded on scoped threads: zstd was 90% of a single-threaded parse.
For 100M float rows the parse went from 1.46s to 0.60s and the SDK's
share of an end-to-end read from 1.57s to 0.90s.

In Python, the sync client's retrieve_datapoints_binary returns objects
implementing the Arrow PyCapsule stream protocol, so polars, pyarrow
and pandas 3 take them without copying. The package adds no runtime
dependency. The offline tests read fixtures written by the platform's
own frame writer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
@JosteinGj

Copy link
Copy Markdown
Contributor Author

Included in #177, which now targets main and carries this PR's commits unchanged.

@JosteinGj JosteinGj closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant