Skip to content

core: avg over decimal declares a parameterized-struct intermediate that deriveIntermediateType cannot evaluate, so EXTENSION_DECLARATION rejects every decimal avg #1239

Description

@nielspardon

FunctionBindingResolver.deriveIntermediateType evaluates a decomposable aggregate's intermediate type expression through TypeExpressionEvaluator. avg over decimal declares its intermediate as a parameterized struct, and ReturnTypeEvaluator has no visit(ParameterizedType.Struct), so it reaches the throwing base and the whole binding is rejected.

functions_arithmetic_decimal.yaml declares:

  - name: "avg"
    impls:
      - args:
          - name: x
            value: "DECIMAL<P,S>"
        nullability: DECLARED_OUTPUT
        decomposable: MANY
        intermediate: "STRUCT<DECIMAL<38,S>,i64>"
        return: "DECIMAL<38,S>"

ParseToPojo.visitStruct only builds a concrete Type.Struct when every field is concrete; DECIMAL<38,S> is a ParameterizedType.Decimal, so the intermediate parses to a ParameterizedType.Struct. The integral and floating-point avg variants declare concrete intermediates (STRUCT<i64,i64>, STRUCT<fp64,i64>) and derive fine, so decimal is the only affected variant — and it is the catalog's only parameterized struct in any position.

Measured on main at fb6a54a:

deriveIntermediateType(avg:dec, [decimal(20,2)])
  => InvalidFunctionBindingException: Cannot derive type for
     FunctionAnchor{urn=extension:io.substrait:functions_arithmetic_decimal, key=avg:dec}:
     Cannot evaluate return-type expression: Struct{nullable=false, fields=[Decimal{nullable=false,
     scale=StringLiteral{nullable=false, value=S}, precision=StringLiteral{nullable=false, value=38}}, I64]}

deriveIntermediateType(avg:i64, [i64])   => Struct{fields=[I64, I64]}
deriveIntermediateType(avg:fp64, [fp64]) => Struct{fields=[FP64, I64]}

Pre-existing, not a regression: the same probe throws identically on main and on #1141's merge result.

Why it matters

ResolvedAggregateBinding.consumesIntermediateState() includes AggregationPhase.UNSPECIFIED, so an ordinary decimal avg measure routes through validateIntermediateSignature and is rejected under AggregateConversion.FunctionBindingValidation.EXTENSION_DECLARATION. That makes it the one remaining derivation gap specific to aggregates, which is what that enum's Javadoc exists to describe — every other function currently named there is a scalar declaration the aggregate validation path never sees.

#1139's census covers return expressions only and has no struct row, so this is not counted in its 86 variants, and ParameterizedReturnTypeTest (added in #1141) filters on returnType() and never inspects intermediate() — so nothing in the build pins the claim either way.

Open question

Binding a parameter of an intermediate expression is not spec-defined. substrait-io/substrait#1151 records the surrounding contradiction and names this exact case as undefined ("How a multi-field intermediate binds. Is avg's STRUCT<i64,i64> one argument of struct type, or two arguments? Nothing says."), and substrait-io/substrait#948 notes the implicit rule that derivation expressions may only use parameters declared in the argument types is nowhere written down. Note S is declared only in the argument DECIMAL<P,S> and the intermediate's first parameter is the literal 38, so the intermediate reads as derived-from-arguments rather than as a binding source; P appears nowhere in the intermediate or the return.

Worth resolving which reading we implement before adding visit(ParameterizedType.Struct), rather than picking one silently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions