Skip to content

fix: [branch-1.2] preserve columnar output when reverting a cached AQE result stage (#6852) - #7650

Merged
andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-6852-branch-1.2
Nov 19, 2026
Merged

andygrove merged 2 commits into
apache:branch-1.2from
andygrove:backport-6852-branch-1.2

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Backport of #6852 to branch-1.2. Closes #6687 on this branch.

Rationale for this change

RevertNativeForTransitionHeavyStages assumed a result stage is always collected as rows. On Spark 4.0+, a cached query whose serializer accepts columnar input, as Comet's ArrowCachedBatchSerializer does, gets a columnar result stage, and once the rule reverted that stage to a row-based root nothing restored the columnar output. Building the cache then failed with FilterExec has column support mismatch (#6687). branch-1.2 has the same rule and the same call sites. Transition reversion is off by default, so only users who enable spark.comet.exec.transitionRevert.enabled are affected. See #6852 for the full analysis.

What changes are included in this PR?

Cherry-pick of d7f4fca (git cherry-pick -x) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes. Both result-stage call sites in RevertNativeForTransitionHeavyStages go through a new revertResultStageIfNeeded, which reads the requested output format off the stage root, and RevertNativeForTransitionHeavyStagesSuite gets the source PR's tests.

Other release branches: branch-1.2 is the newest release branch, so there is no newer one to port to. Whether branch-1.1 needs a backport is not part of this PR.

How are these changes tested?

On this branch, with JDK 17 and the default Spark profile (4.1): ./mvnw test-compile succeeds, and RevertNativeForTransitionHeavyStagesSuite passes (43 tests, 0 failures). No native source changed. With spark/src/main reverted to branch-1.2's version, five of the new tests fail: transition-heavy revert restores columnar output for a cached AQE result stage, the two AQE=true cases of transition-heavy revert of a result root that supports both formats, and both transition-heavy revert keeps the output format of the result root cases. So the branch has the bug and the tests catch it. I did not run the other Spark profiles.

apache#6852)

* fix: preserve columnar output when reverting a cached AQE result stage

RevertNativeForTransitionHeavyStages passed outputColumnar = false for
every result stage. On Spark 4.0+, when the cache serializer accepts
columnar input, as Comet's does, Spark marks a cached AQE plan columnar,
AQE plans its final stage to output columnar batches, and the cache
builder executes it columnar. A reverted row-based root such as
FilterExec then failed with "FilterExec has column support mismatch".

Read the requested format off the stage root instead. Spark's transition
insertion leaves a columnar-only root only when columnar output was
requested, so a row-based reverted root gets a RowToColumnarExec, as
Spark itself would insert.

* fix: skip reverting a result stage whose root supports both formats unless the reverted root does too

* fix: apply the dual-format result-stage check only under AQE

Without AQE the consumer of a result stage reads rows, or, for a cache,
picks the format from the root this rule leaves, so a root that supports
both formats goes back to row output as it does on main. Only AQE fixes
the requested format before it plans the final stage. The check reads
the AQE conf rather than wholePlan, so the plan-only preview under AQE
keeps the same stages that AQE does.

* test: cover the result root's output format at the rule level on every Spark version

The cache test that exercises restoring columnar output for a columnar-only root is skipped before
Spark 4.0, although the branch it covers does not depend on the Spark version. Apply the rule to a
columnar-only root and to a row root directly, and assert the format each keeps.

(cherry picked from commit d7f4fca)

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@andygrove
andygrove merged commit 4d97f4f into apache:branch-1.2 Nov 19, 2026
274 of 276 checks passed
@andygrove
andygrove deleted the backport-6852-branch-1.2 branch November 19, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants