Dedup @task sidecar entries by component identity (v0.1.14)

[Volv-G]

(AI-assisted)

Problem

A pipeline that reaches one shared @task function from several call sites with different task-level options silently ran every one of them on the first call site's configuration.

_build_local_from_python_components keyed the generated <stem>.components.yaml by the hyphenated function name, so a shared run_dbt decorated once with a slim image and once with a fat image produced a single run-dbt entry, and _rewrite_task_componentref_urls recomputed that same name per task — pointing every task at the survivor. Nothing failed: the pipeline compiled, validated, and ran, on the wrong image.

Fix

The name was never an identity. What hydrate regenerates from an entry is determined by the whole local_from_python block — image (explicit or resolved image_id), function, mode, resolve_root, dependencies_from, the persisted unwrap schema, and the source file — so that block, plus the module-qualified function identity, is what dedup keys on.

• One plan, two consumers. _plan_task_sidecar returns TaskSidecarPlan(entries, fragment_by_task); sidecar emission and componentRef rewriting consume the same map, so they cannot drift. A fragment depends on how many distinct components a function generates pipeline-wide, which no per-call-site recomputation can know.
• Module-qualified identity, not __module__. The driver imports pipeline scripts as _tangle_user_pipeline_<uuid>, so runtime __module__ is a fresh string every compile for script-defined tasks and a real dotted name for imported ones — unusable. The namespace is derived from the source layout instead: walk up while __init__.py exists (package_a/tasks.py → package_a.tasks, pkg/__init__.py → pkg), recording the first non-package ancestor relative to the pipeline source dir. __qualname__ rides along.
• Readable names where unambiguous. A base with exactly one identity stays run-dbt (or combine--<schema-hash> when unwrapped), so existing single-component pipelines emit byte-identical sidecars. Once a base collides, every variant is suffixed <base>--<hash> — no arbitrary first-traced winner — and variants are emitted in sorted order so sidecar text is trace-order independent.
• Deterministic, leak-free hashes. SHA-256 prefix over canonical JSON; never hash(), never dict/API ordering. Identity paths are anchored at the pipeline source dir, not the output dir, so fragment names survive project relocation and compiling elsewhere (the hidden submit bundle compiles outside the source tree by design). No image, registry, tag, digest, or path text reaches a fragment name.

Behaviour changes

case	before	after
same function, same config, N calls	1 entry	1 entry (unchanged)
same function, different image/deps/mode/unwrap	1 entry, wrong image for N−1 tasks	N entries, each task on its own
package_a.tasks.run + package_b.tasks.run	hard CompileError	2 entries, both emitted

The removed CompileError ("two distinct @task source files map to the same sidecar fragment") existed to stop one file silently shadowing another; distinct fragments satisfy that intent without rejecting legitimate authoring.

_build_local_from_python_components is deleted, not kept as a wrapper: after this change it had no production caller, and an underscore-private function is not an API worth preserving for downstream tests. _plan_task_sidecar requires identity_root explicitly, because defaulting it to the output directory would quietly reintroduce output-dir-dependent fragment names.

Tests

12 focused tests in tests/test_pipeline_compiler.py, all through the real compile_pipeline surface: same-image dedup; different images → two entries + correct refs + no leakage; image_id resolving to the same ref dedups; dependencies_from; mode/resolve_root; unwrap × image composition; two packages sharing a function name; relocation and different-output-dir fragment stability; call-order determinism (same-file and cross-file); single-variant legacy-name stability.

Manual check

examples/python_pipeline/dedup_image_variants/ — one shared helper decorated three times (slim ×2 identical, fat ×1), with the compile command and expected assertions in its docstring. Verified:

run-dbt--b196ad73a4   image: python:3.12-slim   <- daily_orders, hourly_sessions
run-dbt--6024a73044   image: python:3.12        <- backfill_orders

Recompiling into a different output directory yields the same two fragment names.

Downstream

Discovery's tangle_deploy imports and re-exports the deleted _build_local_from_python_components, and has a test asserting the removed CompileError. Both need cleanup in the submodule pin-bump change, not here — they are correct against the currently pinned commit. The replacement surface is _plan_task_sidecar(...) -> TaskSidecarPlan(entries, fragment_by_task). Coordinated with the Discovery-side owner.

Release

Patch bump 0.1.13 → 0.1.14 (pyproject.toml, tangle_cli/__init__.py fallback, tests/test_packaging.py, uv.lock). Full suite: 1424 passed.