You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
spark: a CTAS round trip drops the target's partitioning and provider #1366
ToSubstraitRel.convertCTAS (ToSubstraitRel.scala:664-676) serializes only the target's names, schema and create mode. Partition columns, bucket spec, provider and location are dropped. ToLogicalPlan.visit(NamedWrite) then builds the table from catalogTable (ToLogicalPlan.scala:613-641), which is always a managed, unpartitioned parquet table at $warehouse/$table. A CTAS round trip therefore creates a different table from the one requested, and reports success.
// CreateDataSourceTableAsSelectCommand for:// CREATE TABLE ctas_part USING orc PARTITIONED BY (p) AS SELECT 1 AS id, 'a' AS p// command.table: partitionColumnNames = [p], provider = orc// -> ToSubstraitRel -> proto -> ToLogicalPlan -> execute
spark.sessionState.catalog.getTableMetadata(TableIdentifier("ctas_part"))
// partitionColumnNames = [], provider = parquet
#1269 added this kind of rejection for InsertIntoHadoopFsRelationCommand and InsertIntoHiveTable, but not for the CTAS commands.
Suggested fix: in convertCTAS, reject a target with partition columns, a bucket spec, or a provider or location that NamedWrite cannot carry, until the target's layout can be represented.
Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing.
Activity