Repository navigation
Accept pandas DataFrames in Python bindings - #8
Merged
Merged
Conversation
…ke frames from Python The store is filled from one decoder, batch_loader, which reads the vendored ArrowSchema/ArrowArray structs from their raw buffers: text in every width and as string views, lists, dates and timestamps, doubles, booleans, integers as text, the null type, and dictionary-encoded columns interned once per used value; sliced arrays and sliced batches are read at their offsets. The parquet loader exports each row group through it, so a file and a frame cannot decode differently, and a table loaded both ways is member-for-member the same store. Arrow C++ is reduced to parquet_io: a file as a C Data stream and a stream as a file. parquet_io_arrow.cpp is the only file including an Arrow header, and CPPLINK_WITH_ARROW (AUTO, ON, OFF) swaps in a stub that fails with the reason, after which the library and the command line link nothing but the C++ standard library. The merged predictions, clusters, waterfalls and the sample are built by BatchBuilder and handed to the writer as the same structs; gen-sample's output is byte-identical and the 1M load takes the same 1.7 s. Python takes a pandas frame, a dict of them, a pyarrow table or reader, or a path, all through __arrow_c_stream__, with object columns sliced on the way in. predict, rescore and cluster return DataFrames, with the reports on last_*; behind each is an ArrowTable exporting borrowed columns, the ids as a dictionary over the store's id arena, zero copy. cluster() reuses the run's rows or resolves a frame or a file through the id index. gen_sample and init work without files. The module built without Arrow installs beside PyPI's pandas and pyarrow 25 and passes the suite; CI gains a no-arrow job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 tasks
4ment
added a commit
that referenced
this pull request
Sep 28, 2026
Read and write every table through the Arrow C Data Interface, and take frames from Python The store is filled from one decoder, batch_loader, which reads the vendored ArrowSchema/ArrowArray structs from their raw buffers: text in every width and as string views, lists, dates and timestamps, doubles, booleans, integers as text, the null type, and dictionary-encoded columns interned once per used value; sliced arrays and sliced batches are read at their offsets. The parquet loader exports each row group through it, so a file and a frame cannot decode differently, and a table loaded both ways is member-for-member the same store. Arrow C++ is reduced to parquet_io: a file as a C Data stream and a stream as a file. parquet_io_arrow.cpp is the only file including an Arrow header, and CPPLINK_WITH_ARROW (AUTO, ON, OFF) swaps in a stub that fails with the reason, after which the library and the command line link nothing but the C++ standard library. The merged predictions, clusters, waterfalls and the sample are built by BatchBuilder and handed to the writer as the same structs; gen-sample's output is byte-identical and the 1M load takes the same 1.7 s. Python takes a pandas frame, a dict of them, a pyarrow table or reader, or a path, all through __arrow_c_stream__, with object columns sliced on the way in. predict, rescore and cluster return DataFrames, with the reports on last_*; behind each is an ArrowTable exporting borrowed columns, the ids as a dictionary over the store's id arena, zero copy. cluster() reuses the run's rows or resolves a frame or a file through the id index. gen_sample and init work without files. The module built without Arrow installs beside PyPI's pandas and pyarrow 25 and passes the suite; CI gains a no-arrow job. * Enhance documentation and examples for parquet support and predictions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read and write every table through the Arrow C Data Interface, and take frames from Python
The store is filled from one decoder, batch_loader, which reads the vendored ArrowSchema/ArrowArray structs from their raw buffers: text in every width and as string views, lists, dates and timestamps, doubles, booleans, integers as text, the null type, and dictionary-encoded columns interned once per used value; sliced arrays and sliced batches are read at their offsets. The parquet loader exports each row group through it, so a file and a frame cannot decode differently, and a table loaded both ways is member-for-member the same store.
Arrow C++ is reduced to parquet_io: a file as a C Data stream and a stream as a file. parquet_io_arrow.cpp is the only file including an Arrow header, and CPPLINK_WITH_ARROW (AUTO, ON, OFF) swaps in a stub that fails with the reason, after which the library and the command line link nothing but the C++ standard library. The merged predictions, clusters, waterfalls and the sample are built by BatchBuilder and handed to the writer as the same structs; gen-sample's output is byte-identical and the 1M load takes the same 1.7 s.
Python takes a pandas frame, a dict of them, a pyarrow table or reader, or a path, all through arrow_c_stream, with object columns sliced on the way in. predict, rescore and cluster return DataFrames, with the reports on last_*; behind each is an ArrowTable exporting borrowed columns, the ids as a dictionary over the store's id arena, zero copy. cluster() reuses the run's rows or resolves a frame or a file through the id index. gen_sample and init work without files. The module built without Arrow installs beside PyPI's pandas and pyarrow 25 and passes the suite; CI gains a no-arrow job.