feat(aggregator): publish snapshot dir to IPNS by Alenar · Pull Request #3622 · IntersectMBO/mithril

Alenar · 2026-10-13T08:56:30Z

Content

This PR add a new IPFS related capability to the aggregator: Publication of the snapshot IPFS MFS folder to IPNS.

This publication requires setting a new ipfs_rpc_server_config.ipns_key_name parameter.

Important

The aggregator does not create the key pair, it must me created beforehand (handled by #3621).

Add a new check on the end-to-end runner: if IPFS is enabled, create a key pair and check that all aggregators keep publishing their snapshot folders to it (by checking that the IPNS published folder contains the archive of the last known immutable).

Also this PR includes several fixes for the ipfs-devnet (most of them are related to the latest change to IPFS maintenance):

ipfs-devnet.sh init: removed download from https://dist.ipfs.tech cdn, leaving only download from github, has it have been winded down.
ipfs-devnet.sh load-webui: switched to download from github releases instead of from the https://ipfs.io/ipfs/ gateway, has it have been winded down.
ipfs-devnet.sh init: add port check at start to prevent starting a swarm if the baked port are not free, prompting the user to kill currently running ipfs (kubo) binaries.
ipfs-devnet.sh start: add a fast readiness check, to ensure that all following commands are executing on live node (a fast enough command could still be running against a stopped node).

Pre-submit checklist

Branch
- Tests are provided (if possible)
- Crates versions are updated (if relevant)
- CHANGELOG file is updated (if relevant)
- Commit sequence broadly makes sense
- Key commits have useful messages
PR
- All check jobs of the CI have succeeded
- Self-reviewed the diff
- Useful pull request description
- Reviewer requested
Documentation
- No new TODOs introduced

Issue(s)

Relates to #3606

github-actions · 2026-10-13T09:11:29Z

Test Results

5 files ± 0 226 suites ±0 48m 57s ⏱️ - 13m 22s
4 011 tests +13 4 011 ✅ +13 0 💤 ±0 0 ❌ ±0
12 674 runs +39 12 674 ✅ +39 0 💤 ±0 0 ❌ ±0

Results for commit d95bd12. ± Comparison against base commit b7a1e69.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

mithril-aggregator ‑ tools::kubo_rpc_client::query::ipfs_add::tests::timeout::timeout_duration_is_capped_at_three_minutes

mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::batch_upload::log_error_as_warning_and_continue_without_failing_if_publishing_to_ipns_fails
mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::batch_upload::publish_directory_cid_to_ipns_using_key_name_if_configured
mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::batch_upload::skip_publishing_to_ipns_without_failing_if_ipns_key_is_not_available
mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::check_ipns_key_availability::key_is_available_if_listed_in_the_ipfs_node_keys
mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::check_ipns_key_availability::key_is_not_available_and_warn_if_listing_the_ipfs_node_keys_fails
mithril-aggregator ‑ file_uploaders::ipfs_uploader::tests::check_ipns_key_availability::key_is_not_available_and_warn_if_not_listed_in_the_ipfs_node_keys
mithril-aggregator ‑ tools::kubo_rpc_client::query::ipfs_add::tests::timeout::timeout_duration_is_capped_at_slow_query_timeout
mithril-aggregator ‑ tools::kubo_rpc_client::query::ipfs_key_ls::tests::return_empty_list_when_keys_are_null
mithril-aggregator ‑ tools::kubo_rpc_client::query::ipfs_key_ls::tests::return_error_if_request_fails
mithril-aggregator ‑ tools::kubo_rpc_client::query::ipfs_key_ls::tests::return_error_if_response_body_is_malformed
…

♻️ This comment has been updated with latest results.

Copilot

🟡 Changes recommended

Correct the Kubo key-listing endpoint and ensure IPNS records are refreshed beyond the 48-hour lifetime.

3 open findings

Use correct Kubo key list RPC endpoint · New
Refresh IPNS records independently of uploads · New
Document public IpnsKey type and fields · New

What changed in this PR

Adds IPNS publication for aggregator snapshot directories, end-to-end validation, and IPFS devnet improvements.

Changes:

Adds configurable Kubo key checks and IPNS publication.
Extends end-to-end scenarios to verify published snapshots.
Improves devnet readiness, port validation, downloads, and Web UI loading.

File	Summary
`mithril-test-lab/mithril-end-to-end/src/toolkit/check/cardano_database.rs`	Verifies IPNS-published immutable archives.
`mithril-test-lab/mithril-end-to-end/src/scenario/minimal.rs`	Supplies IPFS context to checks.
`mithril-test-lab/mithril-end-to-end/src/scenario/full.rs`	Supplies IPFS context to checks.
`mithril-test-lab/mithril-end-to-end/src/mithril/mod.rs`	Defines the snapshot IPNS key name.
`mithril-test-lab/mithril-end-to-end/src/mithril/infrastructure.rs`	Creates test IPNS keys.
`mithril-test-lab/mithril-end-to-end/src/mithril/aggregator.rs`	Configures aggregators for IPNS publication.
`mithril-test-lab/mithril-end-to-end/src/devnet/mod.rs`	Exports IPNS key types.
`mithril-test-lab/mithril-end-to-end/src/devnet/ipfs.rs`	Adds IPNS key and resolution helpers.
`mithril-test-lab/ipfs-devnet/README.md`	Documents devnet changes.
`mithril-test-lab/ipfs-devnet/commands/mkfiles/mkscripts-lifecycle.sh`	Adds node readiness checks.
`mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-download.sh`	Removes deprecated CDN downloads.
`mithril-test-lab/ipfs-devnet/commands/mkfiles/kubo-configure-swarm.sh`	Adds port availability checks.
`mithril-test-lab/ipfs-devnet/commands/ipfs-devnet-load-webui.sh`	Downloads Web UI releases from GitHub.
`mithril-aggregator/src/tools/kubo_rpc_client/query/mod.rs`	Registers new Kubo queries.
`mithril-aggregator/src/tools/kubo_rpc_client/query/ipfs_name_publish.rs`	Implements IPNS publication requests.
`mithril-aggregator/src/tools/kubo_rpc_client/query/ipfs_key_ls.rs`	Implements key listing requests.
`mithril-aggregator/src/tools/kubo_rpc_client/query/ipfs_add.rs`	Uses shared timeout constants.
`mithril-aggregator/src/tools/kubo_rpc_client/mod.rs`	Exports shared timeout constants.
`mithril-aggregator/src/tools/kubo_rpc_client/api.rs`	Defines shared RPC timeouts.
`mithril-aggregator/src/file_uploaders/ipfs_uploader.rs`	Publishes snapshot directories to IPNS.
`mithril-aggregator/src/dependency_injection/builder/protocol/artifacts.rs`	Injects IPNS configuration.
`mithril-aggregator/src/configuration.rs`	Adds IPNS key configuration.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

jpraynaud

LGTM 👍

damrobi

LGTM 👍

turmelclem

LGTM 👍

To allow easy re-use of them in other queries

- Introduced optional `ipns_key_name` to `IpfsUploader`, it enable IPNS publishing if set - Implemented IPNS publishing for directories after batch upload. - Added warning logs for failed IPNS publishing without interrupting operations.

And wire it to the `IpfsUploader`

With the last change in IPFS mainternership (see: https://blog.ipfs.tech/2026-08-beyond-sponsored-gateways/), `dist.ipfs.tech` was closed, as such no CDN downloads are available anymore.

…or Web UI - Replace `ipfs.io` with GitHub release API for fetching Web UI CAR archives, since the `ipfs.io` platform was closed. - Add CID validation post-import to ensure the correct file is loaded, before it was passively handled by the download itself.

A swarm conflict leads to strange bugs and slow networks, this guard should hopefully prevent most of them.

Else requests that requires online node will fail if run very-early.

- Auto create the key with a const name - Check if the key name already exist and continue if yes (useful for 'attached' networks) - Automatically passed to the aggregator config Note: no parameter to disable IPNS publishing if ipfs is enabled nor to change the key name, we may add those later but It would bring much right now.

This will allow to easily ask a specific node to resolve the ipns name (required for the upcoming ipns check). details: - Simplified IPFS devnet command generation using the new `IpfsCommandBuilder`. - Added `IpfsCommand` struct for consistent command execution with improved context handling and support for node index. - Updated `start`, `stop`, and `query` methods to use the new builder and executor. - Streamlined keypair management and node topology initialization logic.

- Added `execute_with_output` method to `IpfsCommand` for execute a command for which we read its outputs. - Enhanced `list_ipns_keys` to return key ID and name as tuples. - New queries: - `get_ipns_key_id`: to fetch IPNS key IDs by name. - `ipns_name_resolve`: to resolve IPNS names to IPFS paths / CID. - `ipfs_resolve`: to resolve IPFS paths to their corresponding CIDs.

- `Aggregator` hold its associated `KuboNode` if IPFS is enabled. - Implemented `aggregator_publish_ipfs_uploaded_artifact_to_ipns` for verifying IPNS publication of artifacts.

To reduce potential flakiness for that check.

…ng to IPNS Not strictly needed has the code was already only throwing a warning if `/v0/name/publish` failed, but allow us to log an error that we control instead of relying on the one thrown by Kubo.

Log the outcomes of batch operations at `debug` level so they are visible without enabling `trace` logs: - batch upload to the IPFS MFS directory finished, with the directory CID - directory published to IPNS Per-file logs stay at `trace` level: a batch can contain thousands of files (~10k immutable files on mainnet), which would flood the logs.

…gator manual

- e2e: scoped IPNS key creation and testing to the leader aggregator only - e2e: removed unused `swarm_dir` from `IpfsDevnet` struct - rustdoc adjustments

Only on the query level, but this allow the IPFS uploader to provide values specific for the mithril snapshot upload usecase. - Add `with_lifetime` and `with_ttl` setters to `IpfsNamePublishQuery`, taking a `std::time::Duration` (sent to kubo in seconds, defaults unchanged: 48h / 5min) - Publish IPNS records from the IPFS uploader with a 5 days lifetime and a 30 minutes TTL

…ON` version * mithril-aggregator from `0.10.20` to `0.10.21` * mithril-end-to-end from `0.5.26` to `0.5.27` * mithril-test-lab/ipfs-devnet/VERSION from `0.1.2` to `0.1.3`

Alenar self-assigned this Oct 13, 2026

Alenar deployed to testing-2-preview October 13, 2026 09:13 — with GitHub Actions Active

Alenar deployed to testing-preview October 13, 2026 09:13 — with GitHub Actions Active

Alenar force-pushed the djo/ipfs/3606/aggregator_publish_snapshot_dir_to_ipns branch from aff42d7 to 11fcbc3 Compare October 13, 2026 21:07

Alenar marked this pull request as ready for review October 13, 2026 21:07

Alenar requested review from damrobi, jpraynaud and turmelclem as code owners October 13, 2026 21:07

Alenar deployed to testing-preview October 13, 2026 21:22 — with GitHub Actions Active

Alenar deployed to testing-2-preview October 13, 2026 21:22 — with GitHub Actions Active

Alenar force-pushed the djo/ipfs/3606/aggregator_publish_snapshot_dir_to_ipns branch from 11fcbc3 to bd4576e Compare October 13, 2026 22:16

Alenar deployed to testing-preview October 13, 2026 22:32 — with GitHub Actions Active

Alenar deployed to testing-2-preview October 13, 2026 22:32 — with GitHub Actions Active

jpraynaud requested a lite review from Copilot October 14, 2026 07:33

Copilot started reviewing on behalf of jpraynaud October 14, 2026 07:33

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Comment thread mithril-aggregator/src/tools/kubo_rpc_client/query/ipfs_key_ls.rs

Comment thread mithril-aggregator/src/tools/kubo_rpc_client/query/ipfs_name_publish.rs Outdated

Comment thread mithril-test-lab/mithril-end-to-end/src/devnet/ipfs.rs

Alenar deployed to testing-preview October 14, 2026 13:58 — with GitHub Actions Active

Alenar deployed to testing-2-preview October 14, 2026 13:58 — with GitHub Actions Active

jpraynaud approved these changes Oct 15, 2026

View reviewed changes

damrobi approved these changes Oct 15, 2026

View reviewed changes

Comment thread mithril-test-lab/mithril-end-to-end/src/devnet/ipfs.rs Outdated

turmelclem approved these changes Oct 15, 2026

View reviewed changes

Alenar added 8 commits October 15, 2026 17:44

refactor(aggregator): centralize timeout constants for IPFS queries

f92e7e5

To allow easy re-use of them in other queries

feat(aggregator): add IPNS publish query

2fe2f6b

feat(aggregator): add configurable ipns_key_name for IPNS publishing

49b2f80

And wire it to the `IpfsUploader`

chore(ipfs-devnet): remove cdn download support

a4340a5

With the last change in IPFS mainternership (see: https://blog.ipfs.tech/2026-08-beyond-sponsored-gateways/), `dist.ipfs.tech` was closed, as such no CDN downloads are available anymore.

feat(ipfs-devnet): add port usage checks to prevent swarm conflicts

f04fd66

A swarm conflict leads to strange bugs and slow networks, this guard should hopefully prevent most of them.

feat(ipfs-devnet): add readiness check with retries for node startup

dcae47f

Else requests that requires online node will fail if run very-early.

Alenar added 11 commits October 15, 2026 17:44

feat(e2e): check IPNS publication in cardano-db scenario

c865328

- `Aggregator` hold its associated `KuboNode` if IPFS is enabled. - Implemented `aggregator_publish_ipfs_uploaded_artifact_to_ipns` for verifying IPNS publication of artifacts.

refactor(e2e): add polling for the check IPNS directory publish

4e90386

To reduce potential flakiness for that check.

feat(aggregator): add IPNS key verification before directory publishi…

7baba6c

…ng to IPNS Not strictly needed has the code was already only throwing a warning if `/v0/name/publish` failed, but allow us to log an error that we control instead of relying on the one thrown by Kubo.

docs(website): add configuration details for IPFS RPC server in aggre…

73631c4

…gator manual

chore: PR reviews adjustments

9925d4c

- e2e: scoped IPNS key creation and testing to the leader aggregator only - e2e: removed unused `swarm_dir` from `IpfsDevnet` struct - rustdoc adjustments

chore: upgrade crate versions and `mithril-test-lab/ipfs-devnet/VERSI…

d95bd12

…ON` version * mithril-aggregator from `0.10.20` to `0.10.21` * mithril-end-to-end from `0.5.26` to `0.5.27` * mithril-test-lab/ipfs-devnet/VERSION from `0.1.2` to `0.1.3`

Alenar force-pushed the djo/ipfs/3606/aggregator_publish_snapshot_dir_to_ipns branch from 7996014 to d95bd12 Compare October 15, 2026 16:15

Alenar deployed to testing-preview October 15, 2026 16:32 — with GitHub Actions Active

Alenar deployed to testing-2-preview October 15, 2026 16:32 — with GitHub Actions Active

Alenar merged commit 4f2f0e3 into main Oct 15, 2026
97 of 98 checks passed

Alenar deleted the djo/ipfs/3606/aggregator_publish_snapshot_dir_to_ipns branch October 15, 2026 16:34

jpraynaud mentioned this pull request Oct 23, 2026

Address the findings of the M1.2 milestone audit #3646

Open

5 tasks done

Navigation Menu

Repository navigation

feat(aggregator): publish snapshot dir to IPNS - #3622

feat(aggregator): publish snapshot dir to IPNS#3622
Alenar merged 19 commits into
mainfrom
djo/ipfs/3606/aggregator_publish_snapshot_dir_to_ipns

Alenar commented Oct 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jpraynaud left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

damrobi left a comment

Uh oh!

Uh oh!

turmelclem left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Alenar commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Content

Pre-submit checklist

Issue(s)

Uh oh!

github-actions Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jpraynaud left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

damrobi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

turmelclem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Alenar commented Oct 13, 2026 •

edited

Loading

github-actions Bot commented Oct 13, 2026 •

edited

Loading