Conversation
Shipping a Python-authored pipeline took two commands and left a mess.
`pipelines compile` wrote `pipeline.yaml`, a `pipeline.components.yaml`
sidecar and sometimes a `pipeline.subgraphs/` directory next to the source,
then `pipeline-runs submit` consumed them. The artifacts are pure build
output, but nothing removed them, so they were committed by accident,
went stale against the script they were generated from, and turned a
one-line change into a two-command ritual that could silently submit
yesterday's graph.
Add `pipeline-runs submit-from-python SCRIPT.py`, which compiles,
hydrates, submits and then removes the bundle -- on dry runs and on
failures too:
tangle sdk pipeline-runs submit-from-python pipeline.py \
--override batch_size=100 \
--image eval-slim=registry.example/img@sha256:... \
--arg shop=acme --annotation owner=team
The two value tiers stay distinct and are documented as such: `--override`
and `--image` are compile-time (`cfg` values and `@task(image_id=...)`
resolution), while `--arg` / `--args-json` / `--arg-secret` are run-time
arguments. Run-tier flags are otherwise identical to `submit`, so the new
command is not a second dialect to learn.
The bundle is compiled beside the script, never in a temp directory. The
compiler resolves the root output path up front and derives every relative
URL from it, so an author-written `ref(url="file://./component.yaml")`
only resolves when the bundle sits in the script's own directory. The
script is resolved physically first, so a symlinked script compiles next
to its real target where its `cfg` file and siblings live. The name comes
from an exclusive create, so two concurrent compiles of one script cannot
collide or overwrite a hand-written sibling YAML, and cleanup is
stem-scoped: only `<stem>.yaml`, `<stem>.components.yaml` and
`<stem>.subgraphs/` are removed, so a neighbour is never caught in it.
Allocation failures are translated into the CLI's own error. Once the
temporary file exists, every exit -- a failed initial close, a compile
error, a failed submit -- passes through best-effort stem-scoped cleanup,
so the original error is what the user sees.
Hydration is forced. The compiled bundle references local sidecars the
server cannot read, so `--no-hydrate` would only produce a run that fails
later and further away.
A multi-entry `--config` prepares *every* entry before creating any run:
each is compiled exactly once, hydrated, merged with its run arguments and
secrets, validated, and frozen into a submit body; only then are the
frozen bodies submitted in order, with no recompilation. A typo or a
broken graph in the last entry therefore creates no runs at all, instead
of landing entry one and failing afterwards. Bundles stay on disk until
the final submit returns, because run-lifecycle hooks are handed the
compiled path and may read it. Submission still goes through the existing
run lifecycle, so submission-id stamping and ambiguous-submit recovery are
unchanged.
Config keys are checked before anything is compiled. A key that merely
looks like a supported field (`overide:`) is rejected rather than dropped,
since dropping it submits a valid run that is not the one authored, while
unrelated keys in a shared config file are left alone. `hydrate:` and
`pipeline_path:` are rejected outright, and the two safety-sensitive
values are type-checked: a non-integer `submit_recovery_attempts` would
crash an ambiguous submit before its recovery lookup, and a string
`trusted_hydration_cli` would quietly enable allow-all hydration.
`--override` / `--image` parsing moves verbatim from `pipelines compile`
into a shared helper so both commands accept the same syntax and reject
the same mistakes. `pipeline-runs submit` is unchanged.
Assisted-By: devx/01a0930b-82ad-779b-afde-8cefb3614443
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shipping a Python-authored pipeline took two commands and left a mess.
pipelines compilewrotepipeline.yaml, apipeline.components.yamlsidecar and sometimes a
pipeline.subgraphs/directory next to the source,then
pipeline-runs submitconsumed them. The artifacts are pure buildoutput, but nothing removed them, so they were committed by accident,
went stale against the script they were generated from, and turned a
one-line change into a two-command ritual that could silently submit
yesterday's graph.
Add
pipeline-runs submit-from-python SCRIPT.py, which compiles,hydrates, submits and then removes the bundle -- on dry runs and on
failures too:
The two value tiers stay distinct and are documented as such:
--overrideand
--imageare compile-time (cfgvalues and@task(image_id=...)resolution), while
--arg/--args-json/--arg-secretare run-timearguments. Run-tier flags are otherwise identical to
submit, so the newcommand is not a second dialect to learn.
The bundle is compiled beside the script, never in a temp directory. The
compiler resolves the root output path up front and derives every relative
URL from it, so an author-written
ref(url="file://./component.yaml")only resolves when the bundle sits in the script's own directory. The
script is resolved physically first, so a symlinked script compiles next
to its real target where its
cfgfile and siblings live. The name comesfrom an exclusive create, so two concurrent compiles of one script cannot
collide or overwrite a hand-written sibling YAML, and cleanup is
stem-scoped: only
<stem>.yaml,<stem>.components.yamland<stem>.subgraphs/are removed, so a neighbour is never caught in it.Allocation failures are translated into the CLI's own error. Once the
temporary file exists, every exit -- a failed initial close, a compile
error, a failed submit -- passes through best-effort stem-scoped cleanup,
so the original error is what the user sees.
Hydration is forced. The compiled bundle references local sidecars the
server cannot read, so
--no-hydratewould only produce a run that failslater and further away.
A multi-entry
--configprepares every entry before creating any run:each is compiled exactly once, hydrated, merged with its run arguments and
secrets, validated, and frozen into a submit body; only then are the
frozen bodies submitted in order, with no recompilation. A typo or a
broken graph in the last entry therefore creates no runs at all, instead
of landing entry one and failing afterwards. Bundles stay on disk until
the final submit returns, because run-lifecycle hooks are handed the
compiled path and may read it. Submission still goes through the existing
run lifecycle, so submission-id stamping and ambiguous-submit recovery are
unchanged.
Config keys are checked before anything is compiled. A key that merely
looks like a supported field (
overide:) is rejected rather than dropped,since dropping it submits a valid run that is not the one authored, while
unrelated keys in a shared config file are left alone.
hydrate:andpipeline_path:are rejected outright, and the two safety-sensitivevalues are type-checked: a non-integer
submit_recovery_attemptswouldcrash an ambiguous submit before its recovery lookup, and a string
trusted_hydration_cliwould quietly enable allow-all hydration.--override/--imageparsing moves verbatim frompipelines compileinto a shared helper so both commands accept the same syntax and reject
the same mistakes.
pipeline-runs submitis unchanged.