Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Milestone
Description
What is the problem the feature request solves?
I would like to start planning ahead for a 1.0.0 release. Let's use this issue to discuss what that means.
Current planned items:
- Demonstrated cost savings for TPC-H and TPC-DS at SF1000 (1TB)
- Consistent logging
- Thorough documentation for compatibility status
- Review all configuration options and consider renaming some for consistency
- Documented policy for how long we will support Spark versions
- Documented process for preventing major performance regressions
- Documented process for versioning (semver) moving forward. What does that mean for Comet?
Nice to have:
- Implement more aggregate expressions
- feat: support approx_count_distinct aggregate expression #4819
- feat: support grouping() and grouping_id() indicator functions #4815
- feat: support listagg / string_agg aggregate (Spark 4.0+) #4816
- feat: support max_by and min_by aggregate expressions #4817
- feat: support Spark HyperLogLog sketch functions (hll_sketch_agg, hll_union_agg, hll_sketch_estimate, hll_union) #4802
- feat: support approx_percentile / percentile_approx aggregate #4801
- feat: support native aggregate function
mode#4782 - feat: support PivotFirst aggregate for the optimized PIVOT fast path #4798
- feat: support kurtosis aggregate #4818
- feat: implement regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy, regr_sxy aggregates #4775
- Implement more window functions
Correctness and critical-priority issues to resolve before 1.0.0
Open issues labeled correctness and/or priority:critical. Items marked 🔴 are also priority:critical. (Generated 2026-07-07)
- 🔴 array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable #4830 array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable
- 🔴 Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results #4813 Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results
- bug: array/map kernels crash on non-null child field via CometLocalTableScanExec #4789 bug: array/map kernels crash on non-null child field via CometLocalTableScanExec
- 🔴 bug:
flattendrops nulls and returns wrong results when a sub-array is null #4788 bug:flattendrops nulls and returns wrong results when a sub-array is null - bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found" #4787 bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found"
- 🔴 bug:
null IN ()returns false instead of null under legacy null-in-empty behavior #4786 bug:null IN ()returns false instead of null under legacy null-in-empty behavior - 🔴 [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) #4764 [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy)
- Documented as a known issue in docs: note invalid UTF-8 scan limitation and FFI import risk #4846
- [Bug] map_from_arrays / map_from_entries do not enforce null-key rejection or spark.sql.mapKeyDedupPolicy #4680 [Bug] map_from_arrays / map_from_entries do not enforce null-key rejection or spark.sql.mapKeyDedupPolicy
- [Bug] CAST(BinaryType AS StringType) uses unsafe from_utf8_unchecked (undefined behaviour) #4488 [Bug] CAST(BinaryType AS StringType) uses unsafe from_utf8_unchecked (undefined behaviour)
Describe the potential solution
No response
Additional context
No response
Reacted by Namgung Chan and Oleks V
Reacted by Bhargava Vadlamani, Oleks V and hsiang-c
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
parthchandra commentedon Jul 6, 2026
I'd like to see iceberg metadata columns support make it #4082 (depends on a bunch of PRs in iceberg-rust though).
mbutrovich commentedon Jul 6, 2026
I will get Iceberg 1.11 support into shape #4840.
feat: add withAlternative alias mechanism for CometConf (#4979)
refactor: remove untested async columnar shuffle (#4985)
andygrove commentedon Jul 28, 2026
We plan on cutting the 1.0.0 release candidate this Friday, 7/31
andygrove commentedon Jul 29, 2026
I would like to backport #5079 to increase TPC-DS coverage
andygrove commentedon Jul 29, 2026
I would also like to back port #5110
andygrove commentedon Jul 29, 2026
another one #5132
andygrove commentedon Jul 31, 2026
We need a few more days for the release. I'm going to stop back porting PRs to branch-1.0 and will do hard reset on main branch next week and then create fresh PRs to bump version, add changeling, etc.
Goal is to have the release candidate available by Friday 8/7.
feat: add withAlternative alias mechanism for CometConf (apache#4979)
refactor: remove untested async columnar shuffle (apache#4985)