Skip to content

Submit Python-authored pipelines in one step (v0.1.12) - #55

Merged
Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/tangle-sdk-pipeline-runs-submit-2415f47
Sep 12, 2026
Merged

Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/tangle-sdk-pipeline-runs-submit-2415f47

Conversation

@Volv-G

@Volv-G Volv-G commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Shipping a Python-authored pipeline took two commands and left a mess.
pipelines compile wrote pipeline.yaml, a pipeline.components.yaml
sidecar and sometimes a pipeline.subgraphs/ directory next to the source,
then pipeline-runs submit consumed them. The artifacts are pure build
output, but nothing removed them, so they were committed by accident,
went stale against the script they were generated from, and turned a
one-line change into a two-command ritual that could silently submit
yesterday's graph.

Add pipeline-runs submit-from-python SCRIPT.py, which compiles,
hydrates, submits and then removes the bundle -- on dry runs and on
failures too:

tangle sdk pipeline-runs submit-from-python pipeline.py \
  --override batch_size=100 \
  --image eval-slim=registry.example/img@sha256:... \
  --arg shop=acme --annotation owner=team

The two value tiers stay distinct and are documented as such: --override
and --image are compile-time (cfg values and @task(image_id=...)
resolution), while --arg / --args-json / --arg-secret are run-time
arguments. Run-tier flags are otherwise identical to submit, so the new
command is not a second dialect to learn.

The bundle is compiled beside the script, never in a temp directory. The
compiler resolves the root output path up front and derives every relative
URL from it, so an author-written ref(url="file://./component.yaml")
only resolves when the bundle sits in the script's own directory. The
script is resolved physically first, so a symlinked script compiles next
to its real target where its cfg file and siblings live. The name comes
from an exclusive create, so two concurrent compiles of one script cannot
collide or overwrite a hand-written sibling YAML, and cleanup is
stem-scoped: only <stem>.yaml, <stem>.components.yaml and
<stem>.subgraphs/ are removed, so a neighbour is never caught in it.
Allocation failures are translated into the CLI's own error. Once the
temporary file exists, every exit -- a failed initial close, a compile
error, a failed submit -- passes through best-effort stem-scoped cleanup,
so the original error is what the user sees.

Hydration is forced. The compiled bundle references local sidecars the
server cannot read, so --no-hydrate would only produce a run that fails
later and further away.

A multi-entry --config prepares every entry before creating any run:
each is compiled exactly once, hydrated, merged with its run arguments and
secrets, validated, and frozen into a submit body; only then are the
frozen bodies submitted in order, with no recompilation. A typo or a
broken graph in the last entry therefore creates no runs at all, instead
of landing entry one and failing afterwards. Bundles stay on disk until
the final submit returns, because run-lifecycle hooks are handed the
compiled path and may read it. Submission still goes through the existing
run lifecycle, so submission-id stamping and ambiguous-submit recovery are
unchanged.

Config keys are checked before anything is compiled. A key that merely
looks like a supported field (overide:) is rejected rather than dropped,
since dropping it submits a valid run that is not the one authored, while
unrelated keys in a shared config file are left alone. hydrate: and
pipeline_path: are rejected outright, and the two safety-sensitive
values are type-checked: a non-integer submit_recovery_attempts would
crash an ambiguous submit before its recovery lookup, and a string
trusted_hydration_cli would quietly enable allow-all hydration.

--override / --image parsing moves verbatim from pipelines compile
into a shared helper so both commands accept the same syntax and reject
the same mistakes. pipeline-runs submit is unchanged.

Shipping a Python-authored pipeline took two commands and left a mess.
`pipelines compile` wrote `pipeline.yaml`, a `pipeline.components.yaml`
sidecar and sometimes a `pipeline.subgraphs/` directory next to the source,
then `pipeline-runs submit` consumed them. The artifacts are pure build
output, but nothing removed them, so they were committed by accident,
went stale against the script they were generated from, and turned a
one-line change into a two-command ritual that could silently submit
yesterday's graph.

Add `pipeline-runs submit-from-python SCRIPT.py`, which compiles,
hydrates, submits and then removes the bundle -- on dry runs and on
failures too:

    tangle sdk pipeline-runs submit-from-python pipeline.py \
      --override batch_size=100 \
      --image eval-slim=registry.example/img@sha256:... \
      --arg shop=acme --annotation owner=team

The two value tiers stay distinct and are documented as such: `--override`
and `--image` are compile-time (`cfg` values and `@task(image_id=...)`
resolution), while `--arg` / `--args-json` / `--arg-secret` are run-time
arguments. Run-tier flags are otherwise identical to `submit`, so the new
command is not a second dialect to learn.

The bundle is compiled beside the script, never in a temp directory. The
compiler resolves the root output path up front and derives every relative
URL from it, so an author-written `ref(url="file://./component.yaml")`
only resolves when the bundle sits in the script's own directory. The
script is resolved physically first, so a symlinked script compiles next
to its real target where its `cfg` file and siblings live. The name comes
from an exclusive create, so two concurrent compiles of one script cannot
collide or overwrite a hand-written sibling YAML, and cleanup is
stem-scoped: only `<stem>.yaml`, `<stem>.components.yaml` and
`<stem>.subgraphs/` are removed, so a neighbour is never caught in it.
Allocation failures are translated into the CLI's own error. Once the
temporary file exists, every exit -- a failed initial close, a compile
error, a failed submit -- passes through best-effort stem-scoped cleanup,
so the original error is what the user sees.

Hydration is forced. The compiled bundle references local sidecars the
server cannot read, so `--no-hydrate` would only produce a run that fails
later and further away.

A multi-entry `--config` prepares *every* entry before creating any run:
each is compiled exactly once, hydrated, merged with its run arguments and
secrets, validated, and frozen into a submit body; only then are the
frozen bodies submitted in order, with no recompilation. A typo or a
broken graph in the last entry therefore creates no runs at all, instead
of landing entry one and failing afterwards. Bundles stay on disk until
the final submit returns, because run-lifecycle hooks are handed the
compiled path and may read it. Submission still goes through the existing
run lifecycle, so submission-id stamping and ambiguous-submit recovery are
unchanged.

Config keys are checked before anything is compiled. A key that merely
looks like a supported field (`overide:`) is rejected rather than dropped,
since dropping it submits a valid run that is not the one authored, while
unrelated keys in a shared config file are left alone. `hydrate:` and
`pipeline_path:` are rejected outright, and the two safety-sensitive
values are type-checked: a non-integer `submit_recovery_attempts` would
crash an ambiguous submit before its recovery lookup, and a string
`trusted_hydration_cli` would quietly enable allow-all hydration.

`--override` / `--image` parsing moves verbatim from `pipelines compile`
into a shared helper so both commands accept the same syntax and reject
the same mistakes. `pipeline-runs submit` is unchanged.

Assisted-By: devx/01a0930b-82ad-779b-afde-8cefb3614443
@Volv-G
Volv-G requested a review from Ark-kun as a code owner September 12, 2026 05:31
@Volv-G
Volv-G merged commit 9d26868 into master Sep 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant