Converting SQL to Substrait, a nullable ROW column loses its fields' NOT NULL:
CREATE TABLE src (c ROW(a INT NOT NULL, b DOUBLE));
SELECT c FROM src;
The root's record type comes back as struct<struct?<i32?, fp64?>>, with a nullable. Measured on main at 4bb02974, through SubstraitCreateStatementParser.processCreateStatementsToCatalog and SqlToSubstrait.
The widening happens in Calcite before isthmus sees the type: SqlDataTypeSpec.deriveType ends in fixUpNullability, which calls createTypeWithNullability, the method that makes a record's fields nullable along with the record. SubstraitCreateStatementParser (line 300) and DdlSqlToRelConverter (line 157) both take the column type from there. This is the Calcite-to-Substrait counterpart of #1154, which #1317 fixed in the other direction.
Calcite does this on purpose: its comment on createTypeWithNullability says that under the SQL standard a nested field of a nullable row is always nullable. Substrait can express the narrower type, so the question is whether isthmus should rebuild the column type with enforceTypeWithNullability to keep what the DDL declared, or accept Calcite's SQL reading.
Converting SQL to Substrait, a nullable
ROWcolumn loses its fields'NOT NULL:The root's record type comes back as
struct<struct?<i32?, fp64?>>, withanullable. Measured onmainat4bb02974, throughSubstraitCreateStatementParser.processCreateStatementsToCatalogandSqlToSubstrait.The widening happens in Calcite before isthmus sees the type:
SqlDataTypeSpec.deriveTypeends infixUpNullability, which callscreateTypeWithNullability, the method that makes a record's fields nullable along with the record.SubstraitCreateStatementParser(line 300) andDdlSqlToRelConverter(line 157) both take the column type from there. This is the Calcite-to-Substrait counterpart of #1154, which #1317 fixed in the other direction.Calcite does this on purpose: its comment on
createTypeWithNullabilitysays that under the SQL standard a nested field of a nullable row is always nullable. Substrait can express the narrower type, so the question is whether isthmus should rebuild the column type withenforceTypeWithNullabilityto keep what the DDL declared, or accept Calcite's SQL reading.