Repository navigation
Support parquet write from Arrow record batch #735
Description
Describe the enhancement requested
I'm working on apache/fluss#107 which enable convert Fluss arow structure data to Parquet directly but found the API missing in here.
Althogh ARROW-11776 supports to write from ArrowReader to file, it read from the ArrowReader, write and close the file direclty. But it's in a very coarse-grained , we almost have no control about the writing. Sometime, we want to control when to close the written parquet. Also it requires ArrowReader, but if the arrow RecordBatch is read continuously from remote server . It's not easy to constuct a ArrowReader.
So, I think we may need to support the interface to write Arrow RecordBatch to Parquet via virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch) = 0
Just to as a show case, the api may look like:
public class ArrowBatchParquetWriter {
void write(RecordBatch recordbatch);
void close()
}
Activity
martin-traverse commentedon Jun 1, 2025
Hello. I have been working on full support for Avro read / write, using the arrow-avro adapter. The intention is to provide a high-level API that allows reading / writing whole files block by block, with each block corresponding to one VSR and supporting VSR recycling.
I'd like to finish the Avro work first. But. Could we do something similar for Parquet? It would be a very simple read / write adapter for whole files, so querying / analytics etc would still want to use the Dataset code path. Of course there is a concern about maintenance overhead, but, I do think this is a gap and a solution would be useful. For example in my project we use Arrow-Java to translate to/from a range of formats, so naturally we want to include Parquet in that. Previously we've gone to the parquet-java API directly and had to build our own internal representation from there - a fair amount of work which would be much better handled in something like Arrow. Also there is the issue of Hadoop dependencies, I know the Parquet guys have been working to eliminate them but there is still quite a bit of fiddling needed avoid pulling those dependencies in - if we could package all that up into an Arrow adapter with a much smaller dependency tree, I think that would be valuable as well.
I'd be happy to sketch out some ideas for review once the Avro work is complete. The parquet-java project already has code for schema translation which helps and somewhere I've got some old code from the last time I looked at this which might save some time as well.
Interested to know people's thoughts on this! In any case there are still one or two commits to go on Avro first.
lidavidm commentedon Jun 2, 2025
I think it's not out of the question, IIRC the concern when I've looked at it with colleagues before at various places is just that parquet-java is row-oriented and you can't get great performance out of it so it wasn't quite worth the effort, but if there's interested contributors I suppose that's not a problem
luoyuxia commentedon Oct 28, 2025
Hi, I'm interested in supporting it. But what do you mean by saying
parquet-java is row-oriented and you can't get great performance out of it? I'm not faimilar withparquet-java. If we just call JNI to convert arrow to parquet, will it bring some permance improvement?luoyuxia commentedon Nov 5, 2025
@lidavidm Hi, I open #904 to support it. I have verified it. Do you have time to review, or anyone can help review my pr? If so, I'll polish my pr to make it ready to review. The pr is almost ready, with some work to make ci pass.