Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions docs/source/contributor-guide/benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,42 @@ These can also be run as a suite on a dedicated machine, see
[Micro Benchmarking on AWS EC2](benchmarking_micro_ec2.md). Published results are in
[benchmarks/results/micro](https://github.com/apache/datafusion-comet/tree/main/benchmarks/results/micro).

## MIN/MAX Reader Filtering

`CometMinMaxBenchmark` compares one `MIN(k)` or `MAX(k)` over an `INT` Parquet column with
`spark.comet.exec.aggregate.dynamicFilter.enabled` disabled (`off`) and enabled (`on`). Run it
with a release build; the make target builds and packages the native library:

```shell
make benchmark-org.apache.spark.sql.benchmark.CometMinMaxBenchmark -- \
1048576 1,4 ascending,descending,random,constant 5 off,on
```

The arguments are total rows, scan partition counts, layouts, minimum measured iterations, and
mode order. The example uses one million rows, one or four files/tasks, and at least five measured iterations per
case. Use `16384 1,4 ascending,descending,random,constant 3 off,on` for a smoke run. The Spark master
is `local[1]` by default; set `COMET_BENCHMARK_MASTER` to change task concurrency and record it
alongside results.

Files use uncompressed Parquet v1 pages with dictionary encoding disabled, so even constant
values reach the row-group size target. Each file has multiple row groups. Ascending input favors
`MIN` and descending input favors `MAX`; the opposite combinations measure overhead when the bound keeps improving. Random input
uses deterministic hashed keys, while constant input measures repeated values. Page-index pruning
and decoder row filtering are disabled to isolate row-group pruning. Only the key is scanned.

Before timing, the suite checks results against Spark, verifies the direct scan/partial aggregate
plan and task count, and requires attachment in every enabled task. It prints scan output rows,
requested data bytes, reader data and metadata bytes, pruned row groups, and attachment counters
from this untimed execution. The timed cases include query planning and result collection, but
exclude data generation and validation.

Repeat with `on,off` to check case-order effects, including JVM warmup and metadata-cache warmth.
Compare elapsed time as well as scan metrics: fewer rows or data bytes do not guarantee a faster
query, especially for small files or when footer inspection and scheduling dominate. Record the
Comet revision, build flags, Spark/JDK versions, machine, master, arguments, and both mode orders
with any reported results. See [MIN/MAX reader filtering](../user-guide/latest/tuning/operators.md#minmax-reader-filtering)
for supported plans and fallback limits.

```{toctree}
:hidden:

Expand Down
26 changes: 25 additions & 1 deletion docs/source/user-guide/latest/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,30 @@ Native aggregates with grouping keys report these additional metrics:
Spill bytes from native sorts, aggregates, and sort-merge joins are also added to Spark's task-level
`diskBytesSpilled` metric in every stage, not only in shuffle stages.

#### MIN/MAX Reader Filtering

When `spark.comet.exec.aggregate.dynamicFilter.enabled=true` is captured in an ungrouped partial
aggregate's plan, its metric map includes these counters. See
[MIN/MAX reader filtering](tuning/operators.md#minmax-reader-filtering) for eligibility and limits.

| Metric | Description |
| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `dynamic_filter_minmax_filters_attached` | Executions that attach their improving MIN/MAX bound to a native Parquet reader. |
| `dynamic_filter_minmax_filters_skipped` | Executions of an eligible MIN/MAX aggregate that skip reader attachment, including those exceeding the footer limits or using encryption. |

Plans rejected before reader attachment is attempted do not increment either counter. An attached
filter can still prune nothing, including when a file's schema adaptation disables pruning.

Reader work remains on the scan. Compare `output_rows`, `bytes_scanned`,
`row_groups_pruned_dynamic_filter`, and `row_groups_pruned_statistics` with filtering disabled.
`bytes_scanned` measures requested data/Bloom-filter ranges, excluding footer and page-index reads;
use `scan_io_metadata_bytes` to include the metadata work of inspecting footers. Row groups pruned
while reading a file increment `row_groups_pruned_dynamic_filter`; pruning when later files open
can instead increment `row_groups_pruned_statistics`. The statistics counter also includes other
predicates, so neither counter alone measures MIN/MAX savings. Page-index and decoder filtering
retain their existing `page_index_rows_pruned` and `pushdown_rows_pruned` counters. There are no
separate evaluated/pruned row counters on the aggregate.

### Hash Joins

With `spark.comet.exec.join.dynamicFilter.enabled=true`, native broadcast and shuffled hash joins
Expand Down Expand Up @@ -231,7 +255,7 @@ execution metrics. Counters accumulate per scan operator; they do not instrument
| `scan_io_metadata_cache_hits` | Successful, cache-eligible metadata opens requiring no storage reads. |
| `scan_io_metadata_cache_misses` | Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
| `scan_io_metadata_prefetches` | Opens that used a load which read storage ahead of them, started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a cache miss. |
| `row_groups_pruned_dynamic_filter` | Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold. |
| `row_groups_pruned_dynamic_filter` | Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold or MIN/MAX bound. |

Reader-level and object-store bytes are two views of the same reads; do not add them together.
Likewise, footer bytes are a subset of metadata bytes, not a third reader-level category. A warm
Expand Down
42 changes: 42 additions & 0 deletions docs/source/user-guide/latest/tuning/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,48 @@ skipping more likely. These settings only tune eligible plans. They cannot enabl
while `spark.comet.exec.aggregate.skipPartial.enabled` is `false`, or for unsupported
accumulators and modes.

## MIN/MAX Reader Filtering

For a Parquet table `events` with a `BIGINT` column `event_id`, enable reader filtering before
planning the query:

```sql
SET spark.comet.exec.aggregate.dynamicFilter.enabled=true;
SELECT MIN(event_id) FROM events;
```

This experimental option is disabled by default. Once a task has observed a non-null minimum,
its native Parquet reader can skip later row groups whose statistics prove that no value is
smaller. `MAX` works the same way with an improving maximum. Each task starts with a fresh bound;
bounds are not shared across Spark partitions or exchanges, or retained for later executions.
The aggregate still computes the result from the rows the reader emits.

Reader filtering applies only to these plans:

| Requirement | Supported scope |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Aggregate | One unfiltered, non-distinct `MIN` or `MAX`, without grouping. Multiple aggregates, including `MIN` and `MAX` together, are excluded. |
| Argument | A direct `TINYINT`, `SMALLINT`, `INT`, or `BIGINT` column. |
| Stage | A partial aggregate consuming input rows. `Final` and `PartialMerge` stages do not produce reader filters. |
| Input | A direct native Parquet scan in the same Comet pipeline. Expressions, filters, limits, standalone projections, Spark exchanges, and JVM/Arrow boundaries between the aggregate and scan stop attachment. Column projection within the reader remains supported. |

Unsupported plans retain ordinary aggregation. Scans with supplied file statistics or encryption
also skip attachment. Before reading rows, Comet inspects the task's file footers. If any file has
more than 64 row groups, or retained decoded footer metadata exceeds 8 MiB, the whole task uses
ordinary aggregation. This inspection adds startup work even when filtering saves no data reads.
For each file, schema adaptation disables pruning if it could hide a conversion error. Missing
statistics, including unknown null counts, can prevent pruning.

Ascending data can establish a useful `MIN` bound early; descending data can do the same for `MAX`.
The opposite ordering can keep improving the bound until the end, leaving few or no row groups to
skip. Random data and constant values exercise different pruning opportunities. Compare filtering
enabled and disabled for your layout and partition count, including query elapsed time, emitted
scan rows, bytes read, and row groups pruned. An attached filter alone does not demonstrate a saving.
See [MIN/MAX metrics](../metrics.md#minmax-reader-filtering) for how to interpret these counters.

Existing Parquet page-index and decoder-filter options can also use the bound. This feature adds
no separate filter over decoded scan batches.

## Local TopK Fusion

Set `spark.comet.exec.topK.fusion.enabled=true` to run an eligible local TopK in the same native
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,14 @@ import java.nio.charset.StandardCharsets
import java.util.Base64

import scala.collection.mutable
import scala.jdk.CollectionConverters._

import org.apache.hadoop.fs.Path
import org.apache.parquet.crypto.DecryptionPropertiesFactory
import org.apache.parquet.crypto.keytools.{KeyToolkit, PropertiesDrivenCryptoFactory}
import org.apache.parquet.crypto.keytools.mocks.InMemoryKMS
import org.apache.parquet.hadoop.ParquetFileReader
import org.apache.parquet.hadoop.util.HadoopInputFile
import org.apache.spark.CometListenerBusUtils
import org.apache.spark.SparkConf
import org.apache.spark.benchmark.Benchmark
Expand All @@ -39,6 +43,7 @@ import org.apache.spark.sql.comet.util.Utils
import org.apache.spark.sql.execution.{ColumnarToRowExec, InputAdapter, LeafExecNode, ProjectExec, QueryExecution, SparkPlan, WholeStageCodegenExec}
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.execution.benchmark.SqlBasedBenchmark
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.DecimalType
import org.apache.spark.sql.util.QueryExecutionListener
Expand Down Expand Up @@ -301,6 +306,49 @@ trait CometBenchmarkBase

protected def isComet(plan: SparkPlan): Boolean = plan.nodeName.startsWith("Comet")

protected val rowGroupPruningScanConfigs: Seq[(String, String)] = Seq(
CometConf.COMET_ENABLED.key -> "true",
CometConf.COMET_EXEC_ENABLED.key -> "true",
CometConf.COMET_NATIVE_SCAN_ENABLED.key -> "true",
CometConf.COMET_BATCH_SIZE.key -> "4096",
CometConf.COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED.key -> "false",
CometConf.COMET_RESPECT_DATAFUSION_CONFIGS.key -> "true",
"spark.comet.datafusion.execution.parquet.enable_page_index" -> "false",
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
SQLConf.LEAF_NODE_DEFAULT_PARALLELISM.key -> "1",
SQLConf.FILES_OPEN_COST_IN_BYTES.key -> "4194304")

protected def inspectParquetFixture(directory: String, expectedFiles: Int): (Long, Seq[Int]) = {
val path = new Path(directory)
val files = path
.getFileSystem(spark.sessionState.newHadoopConf())
.listStatus(path)
.filter(_.getPath.getName.endsWith(".parquet"))
require(
files.length == expectedFiles,
s"Expected $expectedFiles files, found ${files.length}")
val rowGroups = files.toSeq.map { file =>
val reader = ParquetFileReader.open(
HadoopInputFile.fromPath(file.getPath, spark.sessionState.newHadoopConf()))
try reader.getFooter.getBlocks.asScala.size
finally reader.close()
}
require(rowGroups.forall(_ > 1), s"Expected multiple row groups per file: $rowGroups")
(files.map(_.getLen).max, rowGroups)
}

protected def formatParquetReaderMetrics(metrics: Map[String, SQLMetric]): Seq[String] =
Seq(
"output_rows",
"bytes_scanned",
"scan_io_data_bytes",
"scan_io_metadata_bytes",
"row_groups_pruned_dynamic_filter",
"row_groups_pruned_statistics",
"page_index_rows_pruned",
"pushdown_rows_pruned")
.map(name => s"$name=${metrics(name).value}")

protected def addParquetScanCases(
benchmark: Benchmark,
query: String,
Expand Down
Loading