Skip to content

spark: a Hive INSERT carries overwrite in create_mode, which other consumers ignore, so INSERT OVERWRITE can run as an append #1370

Open

Description

@nielspardon

ToSubstraitRel converts an InsertIntoHiveTable to NamedWrite(INSERT) and puts the overwrite flag in create_mode: REPLACE_IF_EXISTS for INSERT OVERWRITE, and ERROR_IF_EXISTS for every plain INSERT (ToSubstraitRel.scala:655). ToLogicalPlan reads REPLACE_IF_EXISTS on an INSERT back as overwrite (ToLogicalPlan.scala:532).

insert into hcm values (1)              -> NamedWrite op=INSERT createMode=ERROR_IF_EXISTS
insert overwrite table hcm values (2)   -> NamedWrite op=INSERT createMode=REPLACE_IF_EXISTS

The spec defines create_mode for CTAS ("Used with CTAS to determine what to do if the table already exists") and does not say what it means on an INSERT. So the overwrite only survives a Spark-to-Spark round trip. Other consumers give it no meaning. In this repo, isthmus' SubstraitRelNodeConverter.visit(NamedWrite) maps INSERT to TableModify.Operation.INSERT without reading create_mode (SubstraitRelNodeConverter.java:1240), so a Spark INSERT OVERWRITE consumed there runs as an append and keeps the old rows. ERROR_IF_EXISTS on a plain insert is also misleading: the table always exists for an insert.

#1269 settled the same question for file writes: create_mode is left unspecified on INSERT, and overwrite is rejected because no INSERT field carries it. Suggested fix: do the same here. Emit UNSPECIFIED for a plain Hive insert and reject INSERT OVERWRITE until the spec can express it, or raise the question upstream if the spec should define create_mode for INSERT.

Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing; the partitioned case was closed by #1269, the unpartitioned one is still open.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions