#1317 stopped isthmus from making a nullable struct's fields nullable when it converts a type to Calcite. Calcite's own type derivation still does it. leastRestrictive (set operations, CASE) and SqlValidatorUtil.deriveJoinRowType on an outer join's null-generating side both go through createTypeWithNullability, which makes a record's fields nullable along with the record. This is the part of #1154 that #1317 left open.
Measured on main at 4bb02974, over a scan with a column s: struct?<i32> (and, for the join, a right side with s: struct<i32>):
|
Substrait |
Calcite |
if_then(k = 1, s, s), converted to Calcite and back |
struct?<i32> in, struct?<i32?> out |
RecordType(INTEGER f0) s0 |
UNION ALL of the scan with itself |
struct?<i32> |
RecordType(INTEGER a) s |
LEFT JOIN, the right side's s |
struct?<i32> |
RecordType(INTEGER a) s |
In the if_then case the loss reaches the Substrait plan on the way back. In the other two it stays on the Calcite side, because a Set and a Join derive their Substrait record types from their Substrait inputs — but the Calcite plan isthmus hands out says less than the Substrait plan it came from.
One option for if_then is to give the CASE its declared Substrait type (rexBuilder.makeCall(type, SqlStdOperatorTable.CASE, args)) instead of letting Calcite derive it. switch would need the same treatment, but there :core's Switch.getType() and Calcite's derived type currently disagree on nullability.
#1317 stopped isthmus from making a nullable struct's fields nullable when it converts a type to Calcite. Calcite's own type derivation still does it.
leastRestrictive(set operations,CASE) andSqlValidatorUtil.deriveJoinRowTypeon an outer join's null-generating side both go throughcreateTypeWithNullability, which makes a record's fields nullable along with the record. This is the part of #1154 that #1317 left open.Measured on
mainat4bb02974, over a scan with a columns: struct?<i32>(and, for the join, a right side withs: struct<i32>):if_then(k = 1, s, s), converted to Calcite and backstruct?<i32>in,struct?<i32?>outRecordType(INTEGER f0) s0UNION ALLof the scan with itselfstruct?<i32>RecordType(INTEGER a) sLEFT JOIN, the right side'ssstruct?<i32>RecordType(INTEGER a) sIn the
if_thencase the loss reaches the Substrait plan on the way back. In the other two it stays on the Calcite side, because aSetand aJoinderive their Substrait record types from their Substrait inputs — but the Calcite plan isthmus hands out says less than the Substrait plan it came from.One option for
if_thenis to give theCASEits declared Substrait type (rexBuilder.makeCall(type, SqlStdOperatorTable.CASE, args)) instead of letting Calcite derive it.switchwould need the same treatment, but there:core'sSwitch.getType()and Calcite's derived type currently disagree on nullability.