Skip to content

spark: a CTAS round trip drops the target's partitioning and provider #1366

Open

Description

@nielspardon

ToSubstraitRel.convertCTAS (ToSubstraitRel.scala:664-676) serializes only the target's names, schema and create mode. Partition columns, bucket spec, provider and location are dropped. ToLogicalPlan.visit(NamedWrite) then builds the table from catalogTable (ToLogicalPlan.scala:613-641), which is always a managed, unpartitioned parquet table at $warehouse/$table. A CTAS round trip therefore creates a different table from the one requested, and reports success.

// CreateDataSourceTableAsSelectCommand for:
//   CREATE TABLE ctas_part USING orc PARTITIONED BY (p) AS SELECT 1 AS id, 'a' AS p
// command.table: partitionColumnNames = [p], provider = orc
// -> ToSubstraitRel -> proto -> ToLogicalPlan -> execute
spark.sessionState.catalog.getTableMetadata(TableIdentifier("ctas_part"))
// partitionColumnNames = [], provider = parquet

#1269 added this kind of rejection for InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable, but not for the CTAS commands.

Suggested fix: in convertCTAS, reject a target with partition columns, a bucket spec, or a provider or location that NamedWrite cannot carry, until the target's layout can be represented.

Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions