You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
spark: a file write drops its format options, so an encrypted Parquet table gets plaintext files and a CSV sep is ignored #1362
ToSubstraitRel.convertFileFormat keeps only five CSV keys and none for Parquet or ORC (ToSubstraitRel.scala:514-533), and ToLogicalPlan.convertFileFormat rebuilds Parquet and ORC writes with options = Map.empty (ToLogicalPlan.scala:490-491). Nothing rejects the options it cannot carry, so a file write that depends on one converts and runs, but produces different files.
Two consequences:
Parquet encryption is silently dropped. With parquet.crypto.factory.class set to PropertiesDrivenCryptoFactory, a direct Spark write with parquet.encryption.column.keys / parquet.encryption.footer.key produces an encrypted file (magic PARE). The round-tripped InsertIntoHadoopFsRelationCommand with the same options writes a plaintext PAR1 file with the protected column readable in the bytes. Reads still succeed, since plaintext files are allowed, so nothing reports the problem.
CSV sep and header are misread. The delimiter is read only from delimiter (line 522), but Spark also accepts sep, and header is compared case-sensitively against "false" (line 525). For CREATE TABLE c (id INT, s STRING) USING csv OPTIONS (sep '|'), the FileHolder carries fieldDelimiter=,, the replayed append writes 2,b, and the table then reads [null,null];[1,a].
spark.sql("CREATE TABLE csv_sep (id INT, s STRING) USING csv OPTIONS (sep '|')")
spark.sql("INSERT INTO csv_sep VALUES (1, 'a')")
// InsertIntoHadoopFsRelationCommand over csv_sep with options = table.storage.properties,// query = SELECT 2 AS id, 'b' AS s -> ToSubstraitRel -> proto -> ToLogicalPlan -> execute
spark.sql("SELECT * FROM csv_sep ORDER BY id") // [null,null];[1,a]
Suggested fix: reject options that FileOrFiles cannot carry (at minimum parquet.encryption.* and ORC's orc.encrypt / orc.mask), read sep before delimiter, and parse boolean options case-insensitively, as Spark does.
Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing; #1269 left the options path unchanged.
The same convertFileFormat is also used to export reads (ToSubstraitRel.scala:671, 674), so both CSV problems affect LocalFiles too, not only writes:
spark.read.option("sep", "\t").schema("id INT, s STRING").csv(tsvDir)
// exported field_delimiter "," -> round trip returns [null,null] for every row
spark.read.option("header", "FALSE").schema("id INT, s STRING").csv(headerlessDir)
// exported header_lines_to_skip 1 -> round trip drops the first data row of each file
Activity
nielspardon commentedon Oct 7, 2026
The same
convertFileFormatis also used to export reads (ToSubstraitRel.scala:671,674), so both CSV problems affectLocalFilestoo, not only writes:Measured on
mainat c130b20 (Spark 3.5.4).