Skip to content

Python: preserve sensitive configuration key precision - #22396

Draft
yoff wants to merge 5 commits into
github:yoff/python-shared-cfg-dataflow-flipfrom
yoff:yoff-python-config-key-precision
Draft

yoff wants to merge 5 commits into
github:yoff/python-shared-cfg-dataflow-flipfrom
yoff:yoff-python-config-key-precision

Conversation

@yoff

@yoff yoff commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Important

Experimental draft stacked on #21925. This must not merge independently of its parent.

The base was revalidated immediately before creation at github/codeql:yoff/python-shared-cfg-dataflow-flip = 1a8e317b4a328bea1059453a2ab3ba6eada09e3f.

Causal evidence

This is not an Airflow model correction. Python's generic SensitiveFunctionCall heuristic marks _get_config_value_from_secret_backend(...) as a secret source because the resolved function name matches the sensitive-name regular expression. Global summaries then preserve the topological return path through Airflow's _get_env_var_option / _get_secret_option, Configuration.get, getlist, and get_mandatory_list_value, but they do not retain the control correlation between (section, key) in self.sensitive_config_values and each caller's concrete arguments.

Consequently, core.EXECUTOR and core.DAGS_FOLDER can inherit the secret source even though those pairs are not members of Airflow's runtime sensitive-key registry and cannot take the secret-backend branches. A secret-backend getter being reachable for some registered keys does not imply that every concrete key handled by the shared getter is sensitive.

The first commit reproduces this with an Airflow-shaped local flow and a passing SPURIOUS expectation. The second commit removes that annotation after the precision fix.

Scope

The change deliberately leaves core dataflow and summary routing unchanged: those paths are topologically valid and useful to other analyses. Instead, SensitiveDataSources identifies a known non-sensitive configuration lookup only when:

  • every resolved target maps conventional section and key parameters;
  • all local sources agree on unique string literals;
  • the receiver, section, and key are the only actual inputs;
  • neither resolved nor syntactic callee names indicate sensitive data; and
  • section/key names do not match sensitive or conservative configuration-secret indicators.

The clear-text logging, clear-text storage, and weak sensitive-data hashing customizations expose those nodes through their existing sanitizer extension points. There are no Airflow names or key values in the implementation.

Exact Airflow evidence

On a locally extracted CodeQL database for apache/airflow@a9da0f7fb48dc7526b2745be3e8fe64e1c775da2:

  • the parent query produced 81 result rows; this branch produces 41;
  • the core.EXECUTOR family at executor_loader.py:225/234 is removed;
  • the core.DAGS_FOLDER family at utils/file.py:68/96 is removed;
  • the final barrier matches 354 call sites / 212 unique concrete pairs; none overlap Airflow's 9 metadata-declared sensitive (section, key) pairs; and
  • evaluation was 11.4s on the parent versus 4.9s on the final query in this local run.

The DCA report's cli_parser.py:64 row is the motivating core.EXECUTOR route. The local extraction did not reproduce that exact sink location, so its evidence remains the DCA comparison plus the independently reproduced shared configuration route; the exact local database did reproduce and remove the related executor and file-path families above.

Conservative controls

Tests retain flow for:

  • concrete PASSWORD, FERNET_KEY, and PASSWORD_FILE selectors;
  • dynamic keys;
  • direct secret-named getter calls; and
  • calls with an additional sensitive fallback argument.

Ambiguous or unresolved calls also remain conservative because every resolved target must satisfy the refinement.

Limitations

This is still a name-based heuristic refinement. It intentionally recognizes only configuration APIs using conventional section / key formal names and does not attempt arbitrary application registry reasoning. Calls with additional value inputs are excluded because the sanitizer blocks the complete lookup result, not one incoming edge. The broad Airflow result delta is why this remains an experimental draft despite the zero-overlap registry audit.

Validation

  • python/ql/test/library-tests/dataflow/sensitive-data
  • python/ql/test/query-tests/Security/CWE-312-CleartextLogging
  • python/ql/test/query-tests/Security/CWE-312-CleartextStorage
  • python/ql/test/query-tests/Security/CWE-312-CleartextStorage-py3
  • python/ql/test/query-tests/Security/CWE-327-WeakSensitiveDataHashing
  • QL formatting checks for all modified QL files

yoff and others added 5 commits August 20, 2026 10:32
Flips the Python dataflow trunk from the legacy CFG (semmle/python/Flow.qll)
and legacy ESSA SSA (semmle/python/essa/*) to the new shared CFG facade
(semmle.python.controlflow.internal.Cfg) and the new SSA adapter
(semmle.python.dataflow.new.internal.SsaImpl), both introduced
additively in the preceding PRs in this stack.

This is the trunk-flip equivalent of the original draft PR github#21894 (kept
around as documentation), rebased on top of the four preparatory PRs:

  P1: Remove AstNode.getAFlowNode() and rewrite callers (github#21919).
  P2: Qualify Flow.qll's AST references with Py:: prefix (github#21920).
  P3: Add new shared-CFG-backed control flow graph (github#21921).
  P4: Add new shared-SSA-backed SSA adapter (github#21923).

The Python dataflow library (semmle/python/dataflow/new/) now imports
the new CFG facade and SSA adapter. All CFG-typed predicates
(ControlFlowNode, CallNode, BasicBlock, NameNode, AttrNode, ...) are
qualified with the Cfg:: prefix; SSA references switch from
EssaVariable/EssaDefinition to SsaImpl::Definition/SourceVariable.

GuardNode is redesigned to use the new CFG's outcome-node model
(isAfterTrue / isAfterFalse) instead of the legacy ConditionBlock +
flipped indirection. Only BarrierGuard<...> is preserved as public
API.

Framework files (Bottle, FastApi, Django, Tornado, Pyramid, Stdlib,
...) are updated to take CFG nodes from the new facade.

A handful of dataflow consistency tweaks for the new CFG:
- Augmented-assignment targets are treated as both load and store.
- 'from X import *' produces uncertain SSA writes for unknown names.
- CFG nodes are canonicalised so dataflow does not see equivalent
  pre/post-order pairs as distinct nodes.

Two AST tweaks for the new CFG:
- AstNodeImpl: omit PEP 695 type-parameter names from
  FunctionDefExpr / ClassDefExpr children.
- ImportResolution: drop the legacy essa import.

Test churn (~175 files): reblessed library- and query-test .expected
files reflect slightly different CFG granularity, different toString
output, and a handful of true alert deltas in security queries.

Verification: all 367 lib + src + consistency-queries compile clean.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The `Cfg::ControlFlowNode` facade re-exports the shared CFG library's
`dominates`/`strictlyDominates` predicates, which are declared
`bindingset[this, that]` + `pragma[inline_late]` and are meant to be used
as bound-pair membership checks. The facade wrappers dropped these
annotations (using plain `pragma[inline]`), so even though the only
callers — the `with` / `async with` taint steps in DataFlowPrivate.qll
and TaintTrackingPrivate.qll — bind both endpoints, the optimizer was
free to materialise `Cfg::ControlFlowNode.strictlyDominates/1` as a full
O(nodes^2) relation over the (larger) shared-CFG node set.

On some projects this dominated analysis time entirely (DCA showed e.g.
ICTU/quality-time and biosimulations regressing ~75-160x). Restoring
`bindingset[this, other]` + `pragma[inline_late]` on the wrappers turns
the predicate back into a bound-pair check and is result-preserving (only
binding annotations change, the predicate body is unchanged).

Reproduced on ICTU/quality-time: full python-security-extended suite went
from stalling >20min on `strictlyDominates` to completing in ~6min; all
ControlFlow and dataflow/coverage library tests pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document the public expression adapter and apply the canonical QL annotation ordering required by the formatter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 529363f5-bc7d-4f0b-9f47-e03ba9aa0cdf
Reproduce the Airflow-shaped loss of section/key identity through layered configuration getters. The concrete non-sensitive key is documented as SPURIOUS while concrete sensitive and dynamic keys remain positive controls.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The sensitive-name heuristic marks calls such as _get_config_value_from_secret_backend as secret sources. Interprocedural summaries then route those sources through every layered configuration getter result without retaining the guard correlation for each concrete section/key pair. This reproduced Airflow clear-text logging false positives for core.EXECUTOR and core.DAGS_FOLDER.

Recognize fully resolved configuration lookups whose only value selectors are unique literal section and key arguments, and expose them as sanitizers to the clear-text logging, clear-text storage, and weak sensitive-data hashing analyses. The implementation checks every resolved target and preserves sensitive callee names and configuration-specific secret indicators.

This belongs in SensitiveDataSources rather than core summary routing: the summarized path is topologically valid, while the missing fact is semantic precision for a name-based sensitive-source heuristic. Core dataflow therefore remains conservative for all other analyses.

Dynamic or ambiguous selectors, sensitive concrete names, direct secret-named getters, and calls with additional value arguments remain flowing. Tests cover PASSWORD, FERNET_KEY, PASSWORD_FILE, dynamic keys, a sensitive fallback, and the Airflow-shaped safe routes. The refinement intentionally relies on conventional section/key parameter names and does not attempt arbitrary application registry reasoning.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant