Why no bulk Arrow→Parquet write API in Java? How to avoid row-by-row RecordConsumer + optimize? #907

Activity

1. Fenil-v

   added

   Type: usageusage questionusage question

   on Nov 6, 2025
2. V-Fenil commented on Nov 11, 2025

   

   V-Fenil

   on Nov 11, 2025

   More actions

   Describe the usage question you have. Please include as many useful details as possible.

   I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

   1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
      → Parquet) that avoids row-by-row calls?
   2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
   3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
   4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

   Component(s)

   Java

   Describe the usage question you have. Please include as many useful details as possible.

   I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

   1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
      → Parquet) that avoids row-by-row calls?
   2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
   3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
   4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

   Component(s)

   Java

   @pitrou any idea on this? Any help would be saver for me
3. raulcd

   transferred this issue fromapache/arrowon Nov 11, 2025
4. Fenil-v commented on Nov 11, 2025

   

   Fenil-v

   on Nov 11, 2025

   Author

   More actions

   Describe the usage question you have. Please include as many useful details as possible.

   I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

   1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
      → Parquet) that avoids row-by-row calls?
   2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
   3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
   4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

   Component(s)

   Java

   Describe the usage question you have. Please include as many useful details as possible.

   I have ~ 20KB objects that I need to write to Parquet efficiently from Java. In C++, C#, and Python there's a direct/bulk Arrow-Parquet write (e.g. WriteTable / write_table) that avoids row-by-row iteration, but in Java I only see row-by-row paths via RecordConsumer or internal/unstable column writers. Questions:

   1. Is there a supported bulk/columnar Arrow-Parquet write API in Java (e.g, VectorSchemaRoot
      → Parquet) that avoids row-by-row calls?
   2. If not, why is Java limited to row-by-row writes today? Any roadmap for feature parity with C++/Python/C#?
   3. For now, what's the recommended optimization path to write 20KB objects at high throughput from Java (without JNI), or is JNI/Dataset the recommended route?
   4. Any best practices (batch sizing, encodings, writer settings) to mitigate the row-by-row overhead?

   Component(s)

   Java

   @pitrou any idea on this? Any help would be saver for me

   @julienledem
5. julienledem commented on Nov 11, 2025

   

   julienledem

   on Nov 11, 2025

   Member

   More actions

   Hello @V-Fenil,
   If a feature is not present, it is usually because it has not been contributed yet. If the internal column writers work for you, you are welcome to propose an API for them.
   For your last question, I would suggest you experiment to see if changing batch size has an effect on what you're trying to achieve.

   👍React with 👍1Reacted by Fenil-vaghasiya
6. lidavidm commented on Nov 11, 2025

   

   lidavidm

   on Nov 11, 2025

   Member

   More actions

   This doesn't even seem to be an Arrow question, given mentions of parquet-java APIs? parquet-java and arrow-java are unrelated projects.

   I say this duplicates #735.

   👍React with 👍2Reacted by Fenil-vaghasiya and Gang Wu
7. lidavidm

   closed this as a duplicateof Support parquet write from Arrow record batch #735 on Nov 11, 2025
8. V-Fenil commented on Nov 12, 2025

   

   V-Fenil

   on Nov 12, 2025

   More actions

   Thanks for re-directing to #735, that was exactly I was looking for.