You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Generating checksum won't be easy after you convert your data to another format. Parquet data types have different representation internally so file bytes will differ.
If you want to ensure completness you can get original bytes of each record:
Then, you can order records by File_Id and Record_Id, and cpmcatenate all values from Record_Bytes. Then, you can collect it as an array of bytes, and then stream is via ByteBufferInputStream into your checksum caculation.
Activity
yruslan commentedon Mar 4, 2025
Generating checksum won't be easy after you convert your data to another format. Parquet data types have different representation internally so file bytes will differ.
If you want to ensure completness you can get original bytes of each record:
Then, you can order records by
File_IdandRecord_Id, and cpmcatenate all values fromRecord_Bytes. Then, you can collect it as an array of bytes, and then stream is viaByteBufferInputStreaminto your checksum caculation.