You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
spark: a Hive INSERT carries overwrite in create_mode, which other consumers ignore, so INSERT OVERWRITE can run as an append #1370
ToSubstraitRel converts an InsertIntoHiveTable to NamedWrite(INSERT) and puts the overwrite flag in create_mode: REPLACE_IF_EXISTS for INSERT OVERWRITE, and ERROR_IF_EXISTS for every plain INSERT (ToSubstraitRel.scala:655). ToLogicalPlan reads REPLACE_IF_EXISTS on an INSERT back as overwrite (ToLogicalPlan.scala:532).
The spec defines create_mode for CTAS ("Used with CTAS to determine what to do if the table already exists") and does not say what it means on an INSERT. So the overwrite only survives a Spark-to-Spark round trip. Other consumers give it no meaning. In this repo, isthmus' SubstraitRelNodeConverter.visit(NamedWrite) maps INSERT to TableModify.Operation.INSERT without reading create_mode (SubstraitRelNodeConverter.java:1240), so a Spark INSERT OVERWRITE consumed there runs as an append and keeps the old rows. ERROR_IF_EXISTS on a plain insert is also misleading: the table always exists for an insert.
#1269 settled the same question for file writes: create_mode is left unspecified on INSERT, and overwrite is rejected because no INSERT field carries it. Suggested fix: do the same here. Emit UNSPECIFIED for a plain Hive insert and reject INSERT OVERWRITE until the spec can express it, or raise the question upstream if the spec should define create_mode for INSERT.
Measured on main at 72e902e (Spark 3.5.4 and 4.0.2). Pre-existing; the partitioned case was closed by #1269, the unpartitioned one is still open.
Activity