Repository navigation
TieringService support convert arrow to parquet directly #437
Description
Search before asking
- I searched in the issues and found nothing similar.
Description
TieringService support convert arrow to parquet directly for higher performance
Willingness to contribute
- I'm willing to submit a PR!
Activity
[-][fluss-lake/fluss-lake-common] Implement Utility classes like Arrow-to-Parquet conversion implementation[/-][+]Implement Utility classes like Arrow-to-Parquet conversion implementation[/+]SteNicholas commentedon Mar 14, 2025
@leonardBang, does this ticket mean that we could introduce utilities to convert arrow to parquet format? IMO, we could reuse the conversion in Iceberg like ArrowReader, ParquetConversions.
wuchong commentedon Mar 14, 2025
@SteNicholas, yes, we can certainly leverage some external libraries or implementations. However, the links you provided focus on converting Parquet to Arrow, whereas our requirement is the reverse: converting Arrow to Parquet.
SteNicholas commentedon Mar 17, 2025
@wuchong, thanks for explanation for the ticket purpose. I would like to do a try to use jni wrapper way to invoke WriteRecordBatch of arrow.
luoyuxia commentedon Apr 30, 2025
I created a issue in arrow-java to track this. Support parquet write from Arrow record batch apache/arrow-java#735
xx789633 commentedon Nov 1, 2025
Hi @luoyuxia , to implement the scenario you described in issue apache/arrow-java#735 (reading Arrow RecordBatches continuously from a remote server and writing to Parquet) using Arrow Java's
DatasetFileWriter, we can subclass ArrowReader and implement a facade over an iterator. For example, arrow jdbc driver demonstrates such approach in its RootArrowReader. Specifically, we need to overrideloadNextBatchinArrowReaderto manually populateVectorSchemaRoot.Regarding the performance comparison between Arrow Java's Parquet writing and the C++ implementation, do we have performance numbers?
luoyuxia commentedon Nov 3, 2025
Just as discuss offline, that's hacky and will require us to implement a new arrow filesystem for the filesystems that not supported by arrow.
luoyuxia commentedon Mar 30, 2026
Splitting this task into two follow-up sub-issues:
RecordBatchRecordBatchto Parquet conversionThis separates the tiering-source read path from the reusable Arrow-to-Parquet conversion utility.
[-]Implement Utility classes like Arrow-to-Parquet conversion implementation[/-][+]TieringService support convert arrow to parquet directly[/+]ci: check tablet server availabitility (apache#437)
ci: check tablet server availabitility (#437)