Skip to content

isthmus: Calcite's own type derivation still makes a nullable struct's fields nullable #1339

Description

@nielspardon

#1317 stopped isthmus from making a nullable struct's fields nullable when it converts a type to Calcite. Calcite's own type derivation still does it. leastRestrictive (set operations, CASE) and SqlValidatorUtil.deriveJoinRowType on an outer join's null-generating side both go through createTypeWithNullability, which makes a record's fields nullable along with the record. This is the part of #1154 that #1317 left open.

Measured on main at 4bb02974, over a scan with a column s: struct?<i32> (and, for the join, a right side with s: struct<i32>):

Substrait Calcite
if_then(k = 1, s, s), converted to Calcite and back struct?<i32> in, struct?<i32?> out RecordType(INTEGER f0) s0
UNION ALL of the scan with itself struct?<i32> RecordType(INTEGER a) s
LEFT JOIN, the right side's s struct?<i32> RecordType(INTEGER a) s

In the if_then case the loss reaches the Substrait plan on the way back. In the other two it stays on the Calcite side, because a Set and a Join derive their Substrait record types from their Substrait inputs — but the Calcite plan isthmus hands out says less than the Substrait plan it came from.

One option for if_then is to give the CASE its declared Substrait type (rexBuilder.makeCall(type, SqlStdOperatorTable.CASE, args)) instead of letting Calcite derive it. switch would need the same treatment, but there :core's Switch.getType() and Calcite's derived type currently disagree on nullability.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions