Skip to content

runtime build: re-key into a directory the build owns, not the dnf install root - #281

Merged
nicksinas merged 2 commits into
mainfrom
nsinas/imx-boot-glob
Sep 18, 2026
Merged

nicksinas merged 2 commits into
mainfrom
nsinas/imx-boot-glob

Conversation

@nicksinas

Copy link
Copy Markdown
Contributor

What

$OUTPUT_DIR is the runtime's dnf install root. The build opened by deleting
$OUTPUT_DIR/imx-boot* to clear a previous build's re-keyed bootloader, so on
i.MX it deleted avocado-img-bootfiles' own /imx-boot and its variants — and
the stone manifest shipped by that same RPM names exactly those files.
stone validate then failed on two files the build had just removed:

[ERROR] Validation failed. 2 file(s) not found:
  device: rootdisk, image: imx_boot_emmc_fastboot
    imx-boot-avocado-imx8mp-evk-sd.bin-flash_evk_emmc_fastboot
  device: rootdisk, image: imx_boot
    imx-boot

It failed on the first build of a fresh project, with no working state to
regress from, and it did not recover. dnf still considers the package
installed, so a reinstall is a no-op, and avocado clean buys exactly one more
build.

Why the glob could not stay

Gating it on re-keying inverts its purpose: it exists for the build after
someone turns fit_key_in_bootloader off. And the re-key wrote over the BSP's
files in place, so once a build had run nothing but dnf could put them back. No
marker file or bookkeeping recovers a file that was overwritten. The output has
to move.

How

  • The re-key script's output directory is now $OUTPUT_DIR/rekeyed, which is
    already its third argument. No feed change.
  • The cleanup is rm -rf "$OUTPUT_DIR/rekeyed", a directory the build owns.
  • That directory goes into AVOCADO_STONE_INCLUDE_PATHS first. The
    avocado-build-<target> hooks append the runtime dir last, so a re-keyed
    image still shadows the BSP's copy of the same name — without the build
    writing into, or deleting from, a directory dnf owns.

stone_include_paths is fixed in the same change, because the above makes the
value multi-path for any project already setting one. The CLI joined with a
space while every avocado-build-<target> hook reads IFS=':', so a second
path reached stone inside one -i argument naming a directory that does not
exist. Silently: one path worked, two did not. The CLI now joins with :, and
its own var-image reader splits on : to match.

Scope

AVOCADO_STONE_INCLUDE_PATHS is now set unconditionally, where it was set only
for a project that configured it. Every target gets -i .../rekeyed whether or
not it ever re-keys; the build script creates the directory first, and a test
pins that ordering so stone is never handed a missing path.

Four existing tests asserted the space-joined value. Their expectations
changed, not the behaviour under them.

provision sets the same variable and is left alone: it consumes built stone
output rather than the runtime directory.

No layer ships rekey-imx-boot.sh today — not meta-avocado, not avocado-os —
and the CLI hard-errors when fit_key_in_bootloader is on without it. So the
re-key path cannot currently run on any target, and this changes the
destination of output nothing produces yet.

Reported against imx8mp-evk; imx93-evk has the same collision. Regression
in #224, present in rc.3, rc.4 and rc.5.

Test

test_create_build_script_* asserts the removal is by directory, that the
directory is recreated after it and exists before it is used as an include
path, and — the general rule this bug argues for — that no rm in the build
script targets a path under $OUTPUT_DIR the build does not own.

stone_include_paths_round_trip_through_the_colon_splitter puts the joined
value through the hooks' splitter and asserts a whitespace reader sees one
path, which was the bug.

cargo fmt --all --check and cargo clippy --all-targets --all-features -D warnings are clean. The suite passes, except 5 tests that need GNU sed and
xargs and fail the same way on an unmodified checkout.

Copilot AI lite review requested due to automatic review settings September 18, 2026 01:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Stone invocation still splits include paths containing spaces because its flag string is expanded unquoted.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Moves re-keyed bootloader outputs into build-owned storage and standardizes Stone include-path handling.

Changes:

  • Re-keyed files now use $OUTPUT_DIR/rekeyed, which is cleaned and prioritized.
  • Include paths now use colon delimiters consistently.
  • Updates tests and changelog documentation.
File summaries
File Description
src/utils/config.rs Colon-joins Stone include paths and updates tests.
src/commands/runtime/var_image.rs Splits include paths on colons.
src/commands/runtime/build.rs Manages the re-keyed output directory and ordering.
CHANGELOG.md Documents the fixes.
Review details
  • Files reviewed: 4/4 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/commands/runtime/var_image.rs
Comment thread src/utils/config.rs

@mobileoverlord mobileoverlord left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diagnosis and the shape of the fix are both right. Moving the output into a
directory the build owns, rather than gating the glob, is the only thing that
recovers a file dnf owns and the build overwrote — and putting it first in the
search order keeps the shadowing the re-key needs. Fixing the separator in the
same change is correct too, since this is what makes the value multi-path for
everyone.

I verified the separator claim against the layers rather than taking it on
faith, and it holds for every hook but two.

avocado-build-jetson-orin-nx and avocado-provision-jetson-orin-nx still
split on whitespace.
In meta-avocado-nvidia, both do:

for path in $AVOCADO_STONE_INCLUDE_PATHS; do

Every other avocado-build-* and avocado-provision-* in the tree uses
IFS=':' read -ra PATHS. I checked all of them; orin-nx is the only pair that
does not, and it came in with the jetson-orin-nx SOM commit rather than being
converted with its siblings.

So this change swaps which target is broken. Today a space-joined multi-path
value works on orin-nx and is broken everywhere else. After this, it is right
everywhere else and one -i naming a nonexistent a:b on orin-nx.

Two things make that worse than the count suggests:

  • It only bites with more than one path, and this PR makes multi-path the norm
    by always prepending the re-key directory. A project that configures nothing
    stays single-path and keeps working, which is exactly the shape that hides.
  • orin-nx is the carrier-as-BSP-extension target, and
    get_stone_include_paths_for_runtime composes a path for every extension
    declaring stone_include_paths. The orin-nx hook's own comment says the
    first entries exist to "let BSP extensions shadow anything (carrier overlay
    files, custom DTBs, etc.)". That shadowing is what stops.

Sequencing matters here. Those hooks ship inside the SDK image, so merging
this takes effect before any layer fix can reach a built SDK. The two-line
change in meta-avocado wants to be in flight now, not after someone reports
silent carrier overlays on orin-nx.

Worth deleting the stale comment at config.rs:5046 in the same pass. It still
tells the next reader the recipe scripts iterate with
for path in $AVOCADO_STONE_INCLUDE_PATHS, which is now true of exactly the
two files that need changing, and is where the space-join assumption came from.

None of that is a reason to hold this. The i.MX failure is a hard, unrecoverable
stop on the first build of a fresh project and is live in rc.3, rc.4 and rc.5;
the orin-nx regression is narrower and needs a project that both runs orin-nx
and configures extension include paths. Land it, and fix the two hooks now.

Nice touch on the general guard — asserting no rm in the build script targets
a path under $OUTPUT_DIR the build does not own is the rule this bug argues
for, and it will catch the next one rather than this one again.

…stall root

## What

`$OUTPUT_DIR` is the runtime's dnf install root. The build opened by deleting
`$OUTPUT_DIR/imx-boot*` to clear a previous build's re-keyed bootloader, so on
i.MX it deleted `avocado-img-bootfiles`' own `/imx-boot` and its variants — and
the stone manifest shipped by that same RPM names exactly those files.
`stone validate` then failed on two files the build had just removed:

```
[ERROR] Validation failed. 2 file(s) not found:
  device: rootdisk, image: imx_boot_emmc_fastboot
    imx-boot-avocado-imx8mp-evk-sd.bin-flash_evk_emmc_fastboot
  device: rootdisk, image: imx_boot
    imx-boot
```

It failed on the first build of a fresh project, with no working state to
regress from, and it did not recover. dnf still considers the package
installed, so a reinstall is a no-op, and `avocado clean` buys exactly one more
build.

## Why the glob could not stay

Gating it on re-keying inverts its purpose: it exists for the build *after*
someone turns `fit_key_in_bootloader` off. And the re-key wrote over the BSP's
files in place, so once a build had run nothing but dnf could put them back. No
marker file or bookkeeping recovers a file that was overwritten. The output has
to move.

## How

- The re-key script's output directory is now `$OUTPUT_DIR/rekeyed`, which is
  already its third argument. No feed change.
- The cleanup is `rm -rf "$OUTPUT_DIR/rekeyed"`, a directory the build owns.
- That directory goes into `AVOCADO_STONE_INCLUDE_PATHS` first. The
  `avocado-build-<target>` hooks append the runtime dir last, so a re-keyed
  image still shadows the BSP's copy of the same name — without the build
  writing into, or deleting from, a directory dnf owns.

`stone_include_paths` is fixed in the same change, because the above makes the
value multi-path for any project already setting one. The CLI joined with a
space while every `avocado-build-<target>` hook reads `IFS=':'`, so a second
path reached stone inside one `-i` argument naming a directory that does not
exist. Silently: one path worked, two did not. The CLI now joins with `:`, and
its own var-image reader splits on `:` to match.

## Scope

`AVOCADO_STONE_INCLUDE_PATHS` is now set unconditionally, where it was set only
for a project that configured it. Every target gets `-i .../rekeyed` whether or
not it ever re-keys; the build script creates the directory first, and a test
pins that ordering so stone is never handed a missing path.

Four existing tests asserted the space-joined value. Their expectations
changed, not the behaviour under them.

`provision` sets the same variable and is left alone: it consumes built stone
output rather than the runtime directory.

No layer ships `rekey-imx-boot.sh` today — not meta-avocado, not avocado-os —
and the CLI hard-errors when `fit_key_in_bootloader` is on without it. So the
re-key path cannot currently run on any target, and this changes the
destination of output nothing produces yet.

Reported against `imx8mp-evk`; `imx93-evk` has the same collision. Regression
in #224, present in rc.3, rc.4 and rc.5.

## Test

`test_create_build_script_*` asserts the removal is by directory, that the
directory is recreated after it and exists before it is used as an include
path, and — the general rule this bug argues for — that no `rm` in the build
script targets a path under `$OUTPUT_DIR` the build does not own.

`stone_include_paths_round_trip_through_the_colon_splitter` puts the joined
value through the hooks' splitter and asserts a whitespace reader sees one
path, which was the bug.

`cargo fmt --all --check` and `cargo clippy --all-targets --all-features -D
warnings` are clean. The suite passes, except 5 tests that need GNU `sed` and
`xargs` and fail the same way on an unmodified checkout.
…from

The rustdoc still described a space-separated return value, and the comment
explaining why entries are absolute still told the next reader that the recipe
scripts iterate with `for path in $AVOCADO_STONE_INCLUDE_PATHS`. That sentence
is where the space join came from, and leaving it would seed the same bug
again.

Both now describe the colon split the hooks actually perform, and the rustdoc
states the limits: a path holding a colon cannot be expressed, and a path
holding a space survives the join but not every reader.

The var-image comment claimed more than the change delivers. Colon separation
fixes multi-path values; it does not rescue a path containing a space, because
the flags collapse into one scalar that the stone call expands unquoted.
Carrying them as an array would fix that, here and in the overlay prepend that
rewrites the same variable. Noted rather than claimed.
@nicksinas

Copy link
Copy Markdown
Contributor Author

Rebased onto main with #279 in it; 91932bf on top.

On the orin-nx pair. Confirmed independently before acting: 2 files split on whitespace, 52 use IFS=':', and after the change none remain. You're right that this swaps which target is broken, and right about why that is worse than 2-of-54 suggests — this PR makes multi-path the norm by always prepending the re-key directory, so the shape that keeps working is exactly the one that hides the problem.

I had flagged avocado-build-jetson-orin-nx as an outlier when I first read the hooks and did not chase it. That was the gap.

The layer fix is written and sitting on nsinas/orin-nx-stone-path-separator in meta-avocado: four loops, two files, converted to the sibling form, bash -n clean, no behaviour change for a single-path value. It is not pushed yet — a cold review against the layer is running first, since read -ra and ${PATHS[@]} assume these scripts run under bash and I want that confirmed against the recipe that installs them rather than assumed. It goes up as soon as that comes back. Agreed it wants to be in flight now rather than after someone reports silent carrier overlays.

On the stale comment at config.rs:5046. Fixed in 91932bf, along with the rustdoc Copilot flagged for the same reason. That comment told the next reader the scripts iterate with for path in $AVOCADO_STONE_INCLUDE_PATHS — which, as you note, is now true of exactly the two files being changed, and is where the space join came from.

One thing I found in the provision hook and deliberately did not fix. avocado-provision-jetson-orin-nx adds -i "$main_input_dir" before the include paths, the reverse of its own build hook and of the other provision hooks I sampled. So orin-nx provision-time shadowing does not work even with the separator corrected. Changing search order changes which file wins for existing users, so it reads as a separate call for whoever owns that BSP rather than something to bundle into a separator fix. Raised it in the cold review as assess-only.

@nicksinas

Copy link
Copy Markdown
Contributor Author

Layer fix is up: avocado-linux/meta-avocado#397. A cold review against the layer came back clean on the diff — both added lines are byte-identical to all 100 existing occurrences, IFS cannot leak (prefix assignment, and the functions only run inside $(...)), no hook sets set -u, and all of them are #!/usr/bin/env bash installed with install -m 0755, so read -ra is safe. No PV/PR bump or checksum needed: file:// SRC_URI plus hash-based task signatures.

Correcting something I said above. I reported the provision hook's ordering — -i "$main_input_dir" ahead of the include paths — as an unexplained divergence worth someone's attention. It is not. Both orin-nx files carry a header explaining the inversion deliberately:

  1. Runtime output dir — the build-time bundle. Contains the composed-and-overridden artifacts produced by avocado build, including the extension-driven carrier-bsp overlay. This must win for everything that's already bundled.

It landed with the SOM target in 89b5872 and is correct as written. Nothing to file, and #397 leaves it alone.

One thing worth adding to both PR bodies. These hooks ship inside the SDK image, so CLI version and hook version are decoupled. On orin-nx with more than one path: post-#281 CLI + pre-#397 SDK gives one bogus -i a:b; pre-#281 CLI + post-#397 SDK gives one bogus -i "a b". The other 49 hooks have carried that second row latent since they were written — #281 is what makes it live. The pairing table is in #397.

@nicksinas
nicksinas merged commit 7df9003 into main Sep 18, 2026
14 of 15 checks passed
@nicksinas
nicksinas deleted the nsinas/imx-boot-glob branch September 18, 2026 02:42
lee-reinhardt added a commit that referenced this pull request Sep 18, 2026
## What

Cut `1.0.0-rc.5`: the `[Unreleased]` changelog section becomes `##
[1.0.0-rc.5] - 2026-09-17`, and the crate version moves from
`1.0.0-rc.4` to `1.0.0-rc.5` in `Cargo.toml` and `Cargo.lock`.

## Release notes

The rc.5 section of `CHANGELOG.md` is the release notes. It covers the
nine PRs merged since rc.4, grouped Changed / Fixed / Security. This is
a bug-fix release for regressions found in the rc.4 phase:

- **Changed:** `avocado install` runs the SDK phase in parallel without
`--force`, with one parallelism rule shared by the install DAG, the
build DAG and the SDK phase (#275).
- **Fixed:** the bare-config `Failed to walk` crash in `ext build`
(#274), the compile-script failure for package-sourced extensions
(#271), the `--dnf-arg` install stamp deletion that made a project
unbuildable (#272), a stamp path that was not one shell word (#272),
`ext build` skipping an extension whose content comes from a
`post_build` hook (#279), the i.MX bootloader deleted by the re-key
cleanup glob plus multi-path `stone_include_paths` (#281), and the
build-time kernel cmdline export deleted by a merge, now covered by the
runtime build stamp (#277).
- **Security:** `cryptoki` 0.12.1 for RUSTSEC-2026-0286 (#278), and the
extension source symlink guard re-armed on the default config path
(#274).

Every bullet that was under `[Unreleased]` on `main` is present; the two
`### Fixed` headings that had accumulated there are merged into one, and
entries were added for the PRs that landed without a changelog line
(#271, #274, #275, #277, #278). An empty `[Unreleased]` heading remains
for the next cycle.

## After merge

Tag the squash commit `1.0.0-rc.5` to run the release workflow. The
`Bump Homebrew tap formula` job will fail as it did for rc.4:
`homebrew-tap` `main` now requires a pull request, so the formula bump
has to be opened by hand from the `SHA256SUMS` asset (rc.4's is still
open as homebrew-tap#1).

## Verification

fmt, clippy (`-D warnings`) and the version guard test pass on this
branch; the version guard reads `1.0.0-rc.5`. `cargo metadata --locked`
agrees with the lockfile.

Smoke test of this exact commit against the `jetson-orin-nano-devkit`
reference project, 2024/edge feed:

- macOS (studio, through the avocado-vm): `avocado clean -f`, `avocado
install -f`, `avocado build` from a fresh volume, all exit 0; rootfs,
initramfs, stone validate/create and the OS bundle produced.
- Linux (atlas, native Docker): `avocado install -f`, `avocado build`,
`avocado provision -r dev --profile tegraflash`, all exit 0; the flash
ran hands-off in under four minutes and the board booted with the built
image's `AVOCADO_OS_BUILD_ID`, all four extensions merged, sshd up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants