Binary datapoints: read, stream, dataframe ingest and compression 2

[JosteinGj]

The whole binary datapoint stack in one PR: binary read (buffered and streamed), dataframe ingest, and compression 2 in both directions. #175 and #176 were closed into it with their commits unchanged, so each commit still reads on its own:

commit	what
edc7392	Read datapoints through the binary endpoint, decoded in parallel (was #175)
0b97418	Stream the binary read frame by frame as it arrives (was #176)
2bebeaf	One table for buffered and streamed reads, long or wide (was #176)
0d62196	Insert a dataframe through the binary endpoint, straight from Arrow
2592a60	Read compression 2 frames, the api's relayed ClickHouse batches
242ca16	Send ingest frames compressed in-band, as compression 2
5c7dfa5	Default binary ingest to zstd level 3
cfd029a	Default binary ingest to compression 2, in-band buffers

Merging only the first commits would put a read on main that understands only compression 1, which an api on the arrow branch never sends.

Binary read

• Rust: TimeSeriesService::retrieve_datapoints_binary sends the ordinary retrieve body to POST /timeseries/data/list/binary and returns one columnar BinaryDatapoints per series. stream_datapoints_binary returns a Stream with one item per frame. Frames are decoded on spawn_blocking, at most min(cores, 8) in flight, kept in order, while the rest of the response downloads. The buffered call collects that stream and builds each series once, at its final size.
• Python (sync client):
  • retrieve_datapoints_binary returns a BinaryReadResult, a sequence of per-series BinaryDatapoints.
  • retrieve_datapoints_binary_stream returns a single-use BinaryDatapointReader.
  • Both offer external_ids, a long-table Arrow stream and to_polars(wide=False|True), all through one helper (_tables.py) so the two cannot drift.
  • Each BinaryDatapoints implements the Arrow PyCapsule stream protocol, so polars, pyarrow and pandas read it without copying. The package adds no runtime dependency.
• Errors:
  • A body without the DHDE end marker was cut short and is a 502, never fewer points.
  • A stream that breaks off ends with an error item after the items that already arrived.
  • A refusal is the Err before there is a stream.
• Differences from the JSON read:
  • There is no cursor (it is a 400); read on by moving start past the last timestamp.
  • A series with no points is absent rather than empty.
  • An unknown series is a 400 for the whole request.
• Aggregates: codec 2 frames come back as avg/min/max/sum. Decimals stay exact, as an unscaled i128 plus a scale.

Dataframe ingest

• Rust: insert_table_binary(&batches, timestamp, &options) takes wide Arrow RecordBatches: a timestamp column plus one column per series, named by external id.
• Python: insert_dataframe_binary(df, timestamp=None, zstd_level=None, compression=None) takes a polars, pandas or pyarrow table through the Arrow C stream.
• Timestamp column: with no timestamp=, it uses the pandas index if that is a timestamp, then a timestamp column, then the only timestamp-typed column.
• Round trip: what to_polars(wide=True) returns goes back in unchanged.
• Values:
  • A null means no point at that timestamp, and pandas exports NaN as null.
  • Unsorted rows are sorted, and a repeated timestamp keeps its last value.
  • Values are stored the way the string path stores them; the offline tests decode both paths' frames and compare.
• Refused before anything is sent:
  • a column that does not name a readable series (404);
  • a timestamp with no time zone, or finer than a millisecond (422);
  • a column its series' value type cannot hold (422).

Compression 2

• Read: the api now relays ClickHouse's compressed batches as compression 2: an Arrow IPC stream whose buffers are zstd-compressed in-band. Both compressions are read. The fixtures are written by the platform's own DatapointFrameWriter.relayStream, and the old compression 1 fixtures are kept as read_*_zstd.bin.
• Ingest:
  • BinaryIngestOptions::compression takes FrameCompression::Buffers (compression 2, the default) or FrameCompression::Payload (compression 1).
  • Buffers is the default because the consumer relays those batches into ClickHouse without decompressing them (platform c9e9351dd). On 100M float points from polars, time until the last point was readable fell from 15.5 s to 13.2 s, at the same cost to the client.
  • In Python it is compression="buffers" | "payload" on the three binary ingest methods, "buffers" by default.
  • At compression 2 the raw length the server recounts is read back from the stream that was written (in_band_raw_len).
  • The ingest side of compression 2 needs platform 357794cce or later.
• Default zstd level: now 3 instead of 9. On noisy floats, level 9 compressed no smaller than level 1 and took several times as long.
• Breaking for Rust callers: BinaryIngestOptions has a new public field, so code that builds it as a struct literal no longer compiles.

Measured

Release build, 100M float rows across 10 series:

	time
read, server alone (compression 2)	2.20 s, down from 6.5 s at compression 1
retrieve_datapoints_binary_stream(...).to_polars()	2.33 s
retrieve_datapoints_binary(...).to_polars()	2.67 s
insert_dataframe_binary from polars	3.1 s, against about 75 s through insert_from_lists_binary

Tests

• Offline (Rust): the Java-written fixtures at both compressions, including several batches per frame and a series cut across frames. Also: frames split at any chunk size, cut-short and bad-frame streams, Buffers frames read back the same as Payload frames, and dataframe-vs-string parity.
• Offline (Python): stand-in servers for the read, the stream and the dataframe round trip, including compression 2 frames.
• Live: test_datapoints_binary_read, the wide polars round trip at both compressions, and a 120k-row pandas round trip.
  • Binary ingest now defaults to compression 2, so every live binary ingest test needs an api that includes 357794cce; an older api refuses the frames with a 400.
  • The Java decoder accepted every compression 2 frame the SDK wrote.

Docs

Tracked in IntelliStream-DataHub/datahub-sdk-docs#115 and written in IntelliStream-DataHub/datahub-sdk-docs#116, stacked on the platform side's #114:

• ingest-timeseries.mdx: the dataframe ingest, the compression option and the zstd default;
• query-and-aggregate.mdx: the binary read;
• the reference pages.

🤖 Generated with Claude Code