Conversation
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll) and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade (semmle.python.controlflow.internal.Cfg) and the new SSA adapter (semmle.python.dataflow.new.internal.SsaImpl), both introduced additively in the preceding PRs in this stack. This is the trunk-flip equivalent of the original draft PR #21894 (kept around as documentation), rebased on top of the four preparatory PRs: P1: Remove AstNode.getAFlowNode() and rewrite callers (#21919). P2: Qualify Flow.qll's AST references with Py:: prefix (#21920). P3: Add new shared-CFG-backed control flow graph (#21921). P4: Add new shared-SSA-backed SSA adapter (#21923). The Python dataflow library (semmle/python/dataflow/new/) now imports the new CFG facade and SSA adapter. All CFG-typed predicates (ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are qualified with the Cfg:: prefix; SSA references switch from EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable. GuardNode is redesigned to use the new CFG's outcome-node model (isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock + flipped indirection. Only BarrierGuard<...> is preserved as public API. Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib, ...) are updated to take CFG nodes from the new facade. A handful of dataflow consistency tweaks for the new CFG: - Augmented-assignment targets are treated as both load and store. - 'from X import *' produces uncertain SSA writes for unknown names. - CFG nodes are canonicalised so dataflow does not see equivalent pre/post-order pairs as distinct nodes. Two AST tweaks for the new CFG: - AstNodeImpl: omit PEP 695 type-parameter names from FunctionDefExpr / ClassDefExpr children. - ImportResolution: drop the legacy essa import. Test churn (~175 files): reblessed library- and query-test .expected files reflect slightly different CFG granularity, different toString output, and a handful of true alert deltas in security queries. Verification: all 367 lib + src + consistency-queries compile clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's `dominates`/`strictlyDominates` predicates, which are declared `bindingset[this, that]` + `pragma[inline_late]` and are meant to be used as bound-pair membership checks. The facade wrappers dropped these annotations (using plain `pragma[inline]`), so even though the only callers — the `with` / `async with` taint steps in DataFlowPrivate.qll and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full O(nodes^2) relation over the (larger) shared-CFG node set. On some projects this dominated analysis time entirely (DCA showed e.g. ICTU/quality-time and biosimulations regressing ~75-160x). Restoring `bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns the predicate back into a bound-pair check and is result-preserving (only binding annotations change, the predicate body is unchanged). Reproduced on ICTU/quality-time: full python-security-extended suite went from stalling >20min on `strictlyDominates` to completing in ~6min; all ControlFlow and dataflow/coverage library tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the public expression adapter and apply the canonical QL annotation ordering required by the formatter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 529363f5-bc7d-4f0b-9f47-e03ba9aa0cdf
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
yoff
force-pushed
the
yoff/python-shared-cfg-dataflow-flip
branch
from
September 22, 2026 11:38
f2a6889 to
f7774ef
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment
Dependent on #21925. This draft does not claim semantic equivalence with legacy ESSA; it measures the closest general shared-SSA gate for
TypeTrackingInput::capturedJumpStep.The gate now requires an actual non-phi
SsaImpl::EssaNodeDefinitionwhosegetSourceVariable()is the capturedSsaSourceVariableand whosegetDefiningNode()is the candidateCfg::DefinitionNode, instead of admitting every syntacticVariable.getAStore().Exact two-commit structure
439edf66b1fcc4b6c73827f1bad0e9d2716db4e8— red-state inline regression test; broad behavior passes with the safe read markedSPURIOUS.33133d6cc5e8cd179d7787454140da64c46220e3— selective shared-SSA implementation; expected lost captures becomeMISSINGand the safe-read spurious results disappear.Tuple measurements
zauberzeug/nicegui@26756ccflocal corpus DBopenai/codex@cd2d84d4local corpus DBqlustered/deepdiff@6f3d5eeblocal corpus DBNo reusable exact Airflow
a9da0f7fb48dc7526b2745be3e8fe64e1c775da2database was present in the current worktree, session artifacts, or known local CodeQL database/cache locations, so this experiment does not fabricate an Airflow measurement.Precision trade-off
The focused test assigns
tracked, reads it through a closure, reassigns the variable to a safe value, then reads through the closure again. The broad gate reports both reads. The SSA gate removes the one broad-only write tuple: three true-positive annotation sites before reassignment are intentionally recorded asMISSING, while twoSPURIOUSannotation sites for the safe sibling read disappear. This is evidence of restored legacy-style selectivity, not proof of semantic equivalence.Validation
codeql test run python/ql/test/library-tests/dataflow/typetracking --threads=2— 2/2 passed in both red and green states.codeql test run python/ql/test/library-tests/dataflow/typetracking python/ql/test/library-tests/dataflow/global-or-captured-vars python/ql/test/library-tests/dataflow/variable-capture python/ql/test/library-tests/dataflow/regression --threads=2— 7/7 passed.codeql query format --check-only python/ql/lib/semmle/python/dataflow/new/internal/TypeTrackingImpl.qll— passed.The branch is ready for a DCA comparison against #21925 (including baseline #38534 / data branch
data/yoff/PR-21925-0-python__1).