Skip to content

fix(arcup): accept a checksum file with no trailing newline - #402

Closed
erkancamli wants to merge 4 commits into
circlefin:mainfrom
erkancamli:fix/arcup-checksum-trailing-newline
Closed

erkancamli wants to merge 4 commits into
circlefin:mainfrom
erkancamli:fix/arcup-checksum-trailing-newline

Conversation

@erkancamli

Copy link
Copy Markdown

What's wrong

verify_checksum_file() decides whether the checksum file is empty from the exit status of read:

if ! read -r expected_checksum expected_name < "$checksum_path"; then
    error "Checksum file is empty: $checksum_path"
fi

read returns non-zero once it hits EOF without seeing a delimiter — even though it has already assigned the variables. So a single-line .sha256 file that simply lacks a trailing newline aborts the install with Checksum file is empty, while the file in fact holds a perfectly good checksum that the very next line would have accepted.

Reproduction

$ cd arc-node
$ ARCUP_SKIP_MAIN=1 && set -- && source arcup/arcup
$ T=$(mktemp -d); printf 'archive bytes' > "$T/a.tar.gz"
$ SUM=$(compute_sha256 "$T/a.tar.gz")
$ NAME=arc-node-v0.8.0-x86_64-unknown-linux-gnu.tar.gz

# with a trailing newline -> accepted
$ printf '%s  %s\n' "$SUM" "$NAME" > "$T/a.sha256"
$ verify_checksum_file "$T/a.tar.gz" "$T/a.sha256" "$NAME" && echo ACCEPTED
ACCEPTED

# byte-identical checksum, only the trailing newline removed -> refused
$ printf '%s  %s' "$SUM" "$NAME" > "$T/b.sha256"
$ verify_checksum_file "$T/a.tar.gz" "$T/b.sha256" "$NAME" && echo ACCEPTED
error: Checksum file is empty: /tmp/.../b.sha256

Scope, honestly

Releases published from this repo are not affected: scripts/release-package.sh writes the file with sha256sum > …, which appends the newline.

It bites the setups arcup deliberately supports through the ARC_REPO and GITHUB_API_URL overrides — mirrors, internal artifact stores, air-gapped relays and hand-generated checksums, where the trailing byte is routinely lost (printf without \n, object-store rewrites, copy-paste). With mainnet approaching, more operators will be pulling binaries through exactly those paths.

The second cost is the diagnostic: the file is not empty, so the operator is sent to look at the wrong thing. A checksum failure during a node install is precisely where a misleading message is expensive.

The fix

Check the parsed value instead of the exit status, and report an unreadable file as unreadable rather than as empty:

if [[ ! -r "$checksum_path" ]]; then
    error "Checksum file is missing or unreadable: $checksum_path"
fi

read -r expected_checksum expected_name < "$checksum_path" || true

if [[ -z "$expected_checksum" ]]; then
    error "Checksum file is empty: $checksum_path"
fi

Nothing is loosened. An empty file, a blank-line file and a bad hash all still fail — the hex/length regex and the filename check downstream are untouched.

Tests

Five cases added to test_checksum_validation in arcup/test_arcup.sh, covering both the fix and the failure paths it must preserve:

  • checksum file without trailing newline passes
  • bare checksum (no filename, no newline) passes
  • empty checksum file fails
  • blank-line checksum file fails
  • missing checksum file fails

Red before, green after — the new tests were run against unpatched arcup/arcup first:

$ git stash push arcup/arcup && bash arcup/test_arcup.sh
ok - valid checksum file passes
ok - checksum filename mismatch fails
error: Checksum file is empty: /tmp/.../archive.tar.gz.sha256
$ echo $?
1

$ git stash pop && bash arcup/test_arcup.sh
... 30 tests, all ok

Full suite: 30/30 passing on bash 5.2 / Linux.

One thing I noticed but left alone

arcup/test_arcup.sh is 800+ lines and is not run by any workflow in .github/workflows/, so this regression test will not actually guard anything until it is wired up. That felt like a separate decision (runner cost, matrix, whether you want the installer gating PRs), so I kept it out of this PR. Happy to send a small follow-up adding it to ci.yml if you want it.

🤖 Generated with Claude Code

https://claude.ai/code/session_011ZRTcjFfRbYTTTCcF87aKC

@erkancamli

Copy link
Copy Markdown
Author

A severity correction on my own PR, so nobody merges this thinking users are hitting it.

The only production source of the checksum file is download_file "$VERSION_TAG" "${ARCHIVE_NAME}.sha256" at line 790, fetched from the GitHub release, and those are produced by scripts/release-package.sh with sha256sum "$ARCHIVE_NAME" > ..., which always emits a trailing newline. So the official path never trips this. It is hardening against a hand crafted or third party checksum file, not a live break.

The underlying bash behaviour is still worth guarding: read -r a b < file on a file with no trailing newline returns failure while still populating both variables, so the old code aborted with "Checksum file is empty" on a file that was neither empty nor malformed.

Also worth flagging: test_arcup.sh is not run by any workflow, so the five new cases will not run automatically. I checked all five workflows. Happy to wire it up in a follow up if that is wanted.

The header comment asks for a bump on any change to this script, and
self_update compares the two versions with version_gt, so an equal
version means every existing install refuses the fix as "already up to
date". Without this the checksum change would ship and reach nobody.
expect_fail only checked that the call failed. The empty, blank-line and
missing-file cases all failed on main too, for the wrong reason, so they
were characterization tests rather than regression tests, and the new
unreadable-file guard had no coverage at all.

expect_fail_message pins the message. With the -r guard removed the
missing-file case now fails and shows the raw bash redirection error plus
an unbound-variable error from set -u, which is what the guard prevents.
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Unsigned Commits Detected

The following commits are missing a verified signature:

  • 26d2f1f by ecamli
  • 00c4319 by ecamli

How to fix: Sign your commits.

@erkancamli

Copy link
Copy Markdown
Author

Two things I got wrong, both now pushed.

The version was not bumped, so this PR could not have reached anyone. arcup/arcup line 7 says:

NOTE: if you make modifications to this script, please increment the version number.
WARNING: the SemVer pattern: major.minor.patch must be followed as we use it to determine if the script is up to date.

and self_update gates on version_gt "$remote_version" "$ARCUP_INSTALLER_VERSION" (lines 423 and 452). With both sides at 0.2.0 every existing install answers "Arcup is already up to date" and refuses the fix. Bumped to 0.2.1.

Three of the five new tests were not testing anything. expect_fail only asserts a non-zero exit, and the empty, blank-line and missing-file cases all fail on main too, for the wrong reasons. So they were characterization tests, and the [[ ! -r ]] guard I added had no coverage at all. expect_fail_message now pins the message. Removing just that guard makes the missing-file case fail with:

expected message: Checksum file is missing or unreadable
actual output:
arcup/arcup: line 662: /tmp/.../absent.sha256: No such file or directory
arcup/arcup: line 664: expected_checksum: unbound variable

which is the real behaviour the guard prevents: a raw redirection error leaked to the user, followed by a set -u abort. That is a better justification for the guard than the one I originally gave, and it is now the tested one.

Correcting my own framing on reachability. The description mentions air-gapped systems and hand-generated checksums. arcup has no offline install mode, so those are not real paths. The honest statement is: the release pipeline (scripts/release-package.sh:31-34) always writes a trailing newline via sha256sum/shasum, so no official release can trigger the original bug. It is reachable only through ARC_REPO pointing at a third-party release, or a mirror via GITHUB_API_URL serving its own assets. On the official path the user-visible win is the -r guard's message, not the newline handling.

30/30 tests pass on bash 5.2. Happy to split the -r guard into its own PR if you would rather keep this one to the newline change.

@erkancamli

Copy link
Copy Markdown
Author

Closing this myself. I should have read CONTRIBUTING.md before opening it.

The policy is explicit: this repo does not accept unsolicited contributions, a PR must be preceded by an issue and an assignment, and PRs that skip that are closed without review. This one skipped it, and so did the replacement I briefly opened as #418, which the bot closed for exactly that reason. That is my mistake twice over and I would rather withdraw than keep tripping the same rule.

Two things are worth leaving in the thread in case they are useful to a maintainer.

The bug is real. verify_checksum_file (arcup/arcup:659) decides whether the checksum file is empty from the exit status of read, which returns non-zero at EOF even when it populated the variables. On bash 5.2.21:

$ printf 'abc  name' > nl.txt
$ if ! read -r a b < nl.txt; then echo "READ FAILED, a=[$a] b=[$b]"; fi
READ FAILED, a=[abc] b=[name]

So a single-line checksum file with no trailing newline is rejected as empty. Separately, a missing or unreadable checksum path leaks a raw redirection error and then aborts on an unbound variable under set -u.

No official release can trigger the first one. scripts/release-package.sh:31-34 writes the file with sha256sum or shasum -a 256, both of which always emit a trailing newline, and arcup has no offline install mode. It is reachable through ARC_REPO pointing at a third-party release, or a mirror via GITHUB_API_URL. The clearer user-facing win is the second one, the error message.

I am happy to open an issue for it and wait for assignment if that is welcome, but I will not open another PR here unless someone asks me to. Sorry for the noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant