Skip to content

spark: a file write drops its format options, so an encrypted Parquet table gets plaintext files and a CSV sep is ignored #1362

Open

Description

@nielspardon

ToSubstraitRel.convertFileFormat keeps only five CSV keys and none for Parquet or ORC (ToSubstraitRel.scala:514-533), and ToLogicalPlan.convertFileFormat rebuilds Parquet and ORC writes with options = Map.empty (ToLogicalPlan.scala:490-491). Nothing rejects the options it cannot carry, so a file write that depends on one converts and runs, but produces different files.

Two consequences:

  • Parquet encryption is silently dropped. With parquet.crypto.factory.class set to PropertiesDrivenCryptoFactory, a direct Spark write with parquet.encryption.column.keys / parquet.encryption.footer.key produces an encrypted file (magic PARE). The round-tripped InsertIntoHadoopFsRelationCommand with the same options writes a plaintext PAR1 file with the protected column readable in the bytes. Reads still succeed, since plaintext files are allowed, so nothing reports the problem.
  • CSV sep and header are misread. The delimiter is read only from delimiter (line 522), but Spark also accepts sep, and header is compared case-sensitively against "false" (line 525). For CREATE TABLE c (id INT, s STRING) USING csv OPTIONS (sep '|'), the FileHolder carries fieldDelimiter=,, the replayed append writes 2,b, and the table then reads [null,null];[1,a].
spark.sql("CREATE TABLE csv_sep (id INT, s STRING) USING csv OPTIONS (sep '|')")
spark.sql("INSERT INTO csv_sep VALUES (1, 'a')")
// InsertIntoHadoopFsRelationCommand over csv_sep with options = table.storage.properties,
// query = SELECT 2 AS id, 'b' AS s -> ToSubstraitRel -> proto -> ToLogicalPlan -> execute
spark.sql("SELECT * FROM csv_sep ORDER BY id")  // [null,null];[1,a]

Suggested fix: reject options that FileOrFiles cannot carry (at minimum parquet.encryption.* and ORC's orc.encrypt / orc.mask), read sep before delimiter, and parse boolean options case-insensitively, as Spark does.

Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing; #1269 left the options path unchanged.

Activity

  1. nielspardon commented on Oct 7, 2026

    @nielspardon
    MemberAuthor

    The same convertFileFormat is also used to export reads (ToSubstraitRel.scala:671, 674), so both CSV problems affect LocalFiles too, not only writes:

    spark.read.option("sep", "\t").schema("id INT, s STRING").csv(tsvDir)
    // exported field_delimiter "," -> round trip returns [null,null] for every row
    spark.read.option("header", "FALSE").schema("id INT, s STRING").csv(headerlessDir)
    // exported header_lines_to_skip 1 -> round trip drops the first data row of each file

    Measured on main at c130b20 (Spark 3.5.4).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions