Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/source/contributor-guide/expression-audits/agg_funcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,15 @@
- Performance (tuned 2026-09-09, PR [#5803](https://github.com/apache/datafusion-comet/pull/5803)): as for [collect_list](#collect-list), grouped `collect_set` no longer goes through `GroupsAccumulatorAdapter`. `CollectSetGroupsAccumulator` encodes each batch once for all of its groups and deduplicates against an open-addressed index over a shared arena, replacing one hash table plus one owned `Row` per distinct value per group. 60-93% faster ([#5797](https://github.com/apache/datafusion-comet/issues/5797)). Benchmark: `benches/collect.rs`.
- `CometCollectSet` reports `Incompatible` for float and double input when `spark.comet.exec.strictFloatingPoint=true`, because the native distinct comparison treats `NaN == NaN` and collapses repeated `NaN`s into a single element while Spark keeps each one. The native path for floating-point input is then opt-in via `spark.comet.expression.CollectSet.allowIncompatible=true`. All other input types are `Compatible`.

## kurtosis

- Spark 3.4.3 (audited 2026-07-03): `Kurtosis(child, nullOnDivideByZero)` extends `CentralMomentAgg` with `momentOrder = 4`. Excess kurtosis (Fisher), formula `n * m4 / (m2 * m2) - 3.0`; empty group → `NULL`; `m2 == 0` → `NULL` when `nullOnDivideByZero=true` (default when `spark.sql.legacy.statisticalAggregate=false`) else `NaN`. Any numeric input is cast to `Double` by `ImplicitCastInputTypes`.
- Spark 3.5.8 (audited 2026-07-03): identical to 3.4.3.
- Spark 4.0.1 (audited 2026-07-03): identical to 3.4.3. No collation involvement.
- Spark 4.1.1 (audited 2026-07-03): identical to 3.4.3.
- `CometKurtosis` maps the aggregate to a Comet-owned `Kurtosis` UDAF whose intermediate state (`[n, avg, m2, m3, m4]` Float64) mirrors Spark's `CentralMomentAgg` buffer for `momentOrder = 4`, so Partial output produced by either engine has the same wire format. The Rust update/merge kernels are a direct port of Spark's `updateExpressionsDef` and `mergeExpressions`. `supportsSparkPartialToNativeFinal` and `supportsNativePartialToSparkFinal` are left at the default `false`, matching the conservative policy the other `CentralMomentAgg` serdes (`Variance`, `Stddev`) already use in the same file.
- Window use (`kurtosis(x) OVER (...)`) currently falls back to Spark: the window path doesn't wire the Comet aggregate for kurtosis today.

## listagg

- Spark 3.4.3 (audited 2026-07-03): does not exist. `ListAgg` was added in Spark 4.0.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/expressions.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The tables below list every Spark built-in expression with its current status.
| `grouping_id` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS |
| `hll_sketch_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) |
| `hll_union_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) |
| `kurtosis` | 🔜 | — | Not yet implemented natively |
| `kurtosis` | ✅ | Native | Excess kurtosis (Fisher definition). |
| `last` | ✅ | Native | |
| `last_value` | ✅ | Native | |
| `listagg` | ✅ | Native | Spark 4.0+. `StringType` input with a literal delimiter; `WITHIN GROUP (ORDER BY ...)` and `BinaryType` inputs fall back to Spark. Without `WITHIN GROUP`, concatenation order is the (non-deterministic) group arrival order, matching Spark. |
Expand Down
15 changes: 13 additions & 2 deletions native/core/src/execution/planner/aggregate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ use datafusion_comet_proto::{
};
use datafusion_comet_spark_expr::{
ApproxPercentile, Avg, AvgDecimal, BloomFilterAgg, Cast, CometCollectList, CometCollectSet,
Correlation, Covariance, EvalMode, HllPlusPlus, HllSketchAgg, HllUnionAgg, MaxMinBy, Mode,
Regr, RegrType, SparkBloomFilterVersion, SparkCastOptions, SparkListAgg, SparkMinMax,
Correlation, Covariance, EvalMode, HllPlusPlus, HllSketchAgg, HllUnionAgg, Kurtosis, MaxMinBy,
Mode, Regr, RegrType, SparkBloomFilterVersion, SparkCastOptions, SparkListAgg, SparkMinMax,
SparkPercentile, SparkPivotFirst, SparkStructMinMax, Stddev, SumDecimal, SumInteger, Variance,
ZeroOrder,
};
Expand Down Expand Up @@ -622,6 +622,17 @@ impl PhysicalPlanner {
))),
}
}
AggExprStruct::Kurtosis(expr) => {
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
let func = AggregateUDF::new_from_impl(Kurtosis::new(
"kurtosis",
expr.null_on_divide_by_zero,
from_protobuf_eval_mode(expr.eval_mode)?,
spark_expr.expr_id,
Arc::clone(&self.query_context_registry),
));
Self::create_aggr_func_expr("kurtosis", schema, vec![child], func)
}
AggExprStruct::Stddev(expr) => {
let child = self.create_expr(expr.child.as_ref().unwrap(), Arc::clone(&schema))?;
let datatype = to_arrow_datatype(expr.datatype.as_ref().unwrap());
Expand Down
15 changes: 15 additions & 0 deletions native/proto/src/proto/expr.proto
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ message AggExpr {
HllSketchAgg hllSketchAgg = 27;
HllUnionAgg hllUnionAgg = 28;
PivotFirst pivotFirst = 29;
Kurtosis kurtosis = 30;
}

// Optional filter expression for SQL FILTER (WHERE ...) clause.
Expand Down Expand Up @@ -266,6 +267,20 @@ message Stddev {
StatisticsType stats_type = 4;
}

// Excess kurtosis (Fisher definition: normal distribution -> 0). Spark's
// intermediate buffer is `[n, avg, m2, m3, m4]` of Float64 to match the
// `CentralMomentAgg` DeclarativeAggregate wire format for mixed
// Partial/Final execution with Spark.
message Kurtosis {
Expr child = 1;
bool null_on_divide_by_zero = 2;
// The mode of the `Divide` in Spark's final `n * m4 / (m2 * m2) - 3.0`, captured when the
// aggregate was built. A divisor that underflows to zero returns null in LEGACY mode and
// raises DIVIDE_BY_ZERO in ANSI mode. That is a different switch from
// `null_on_divide_by_zero`, which comes from `legacyStatisticalAggregate`.
EvalMode eval_mode = 3;
}

message Correlation {
Expr child1 = 1;
Expr child2 = 2;
Expand Down
Loading