A prediction, written before the work, so that it can be wrong.
Written 2026-09-28, from measurements taken while serving the first real snapshot sync this node has ever performed (#184). The point of writing it now is that afterwards nobody can tell a good estimate from a lucky one.
Companion to snapshot-sync-estimates.md, and written for the same reason.
Everything in §2 was measured on this machine, server and client on separate physical disks, over loopback. Everything in §4 and §5 is an estimate built on those measurements, and each one names the assumption it rests on.
One correction carried forward: an earlier note in #185 and #166 put the read amplification at 20×. That was wrong — it compared disk reads over an 860 s window against bytes delivered over a 360 s window. Measured over a single window it is 6.5×. Everything below uses the corrected figure.
Serving headers to one syncing client:
| before #185 | after #185 | |
|---|---|---|
| throughput | 934 KB/s | 5,600–7,100 KB/s |
| server CPU | 12% of one core | 54% |
| client CPU | ~0% | 38% |
| server disk read | 6.5 MB/s | 37.4 MB/s |
| amplification | 7.0× | 6.5× |
| queue depth | 1 | 16 |
#185 changed when the reads happen, not what is read: amplification is unchanged, throughput is 7× higher. That is what raising queue depth does, and it is worth stating plainly because the freezer attacks the other half.
~7.1 KB of disk is read to deliver 1.1 KB of header.
Headers and bodies are keyed by hash (CF_HEADERS, CF_BODIES). Hashes
are uniformly distributed, so two consecutive blocks share neither a data block
nor usually a file. Every lookup is an independent descent: consult the table
index, test the bloom filter, fetch and decompress a whole data block, return
1.1 KB from it.
Storing by number in an append-only file removes the cause rather than working around it. Consecutive blocks become consecutive bytes.
A header request carries 192 hashes' worth of chain (BlockHeadersRequest
walks 192 back from one hash). Under a freezer that is one contiguous ~211 KB
read instead of 192 scattered ones.
| now | predicted | |
|---|---|---|
| amplification | 6.5× | ~1.05× |
| disk read at today's output | 37.4 MB/s | ~6 MB/s |
| random reads/s | ~5,300 | ~40 |
Prediction: the disk stops being the constraint, and throughput rises until CPU binds.
Server CPU is 54% of one core for 5,643 KB/s, on two pinned cores → ~20 MB/s. Client CPU is 38% → ~29 MB/s. So the server binds first:
~20 MB/s, about 3.7× beyond today, ~22× over the original. The header phase of a snapshot sync: 3 hours → 25 minutes → ~8 minutes.
The assumption this rests on, stated so it can be checked: that the disk is what binds today. That is inferred from CPU headroom on both sides while 37 MB/s of random reads are in flight — not from benchmarking the device. If the real limit is the 16-thread pool or lock contention, the freezer buys less. Measuring the device's random-vs-sequential capability would settle it, and has not been done.
Bodies have the same storage problem and will not get the same benefit, because the protocol asks for them differently:
pub struct BodyRequest { pub id: u64, pub hash: B256 } // ONE hashOne body per request, against 192 headers per request. So a freezer cannot collapse a body range into one read the way it can for headers — each body is still a separate request, a separate lookup, a separate round trip.
What is still gained:
- No LSM overhead per lookup. No bloom test, no index descent, no block decompression to extract one value. A direct offset read.
- Readahead works. Consecutive bodies are adjacent in the file, so fetching block N pulls N+1…N+k into page cache for free. During a sequential sync that is most of the next requests already resident.
Prediction: 2–3× on body serving, against ~6.5× for headers. Lower confidence than the header figure — body sizes vary with transaction count in a way header sizes do not, and the readahead benefit depends on the client requesting in order, which it does during a sync and may not otherwise.
The protocol shape, not the storage, is what limits the body case. A batched body request — n bodies by range, the way headers already work — would let a freezer answer a whole range in one contiguous read, and would cut round trips by the same factor. That is a wire-protocol change and therefore an rskj compatibility question, but it is where the remaining win is.
Without it, the freezer's body gain is capped by one round trip per block whatever the disk does.
Headers in one file, bodies in another — not header-body-header-body.
Measured on mainnet blocks (#4,194,304 through #9,240,576): a whole block is 1,534–3,561 bytes, averaging ~2,500. The header is ~1,100 of that, so a body averages ~1,400.
A 192-header request therefore spans:
| layout | bytes covered | to deliver |
|---|---|---|
| headers only | 211 KB | 211 KB |
| interleaved | 480 KB | 211 KB |
Interleaving reads 2.3× more for the same answer. That is most of the amplification the freezer exists to remove, put straight back — and it would be paid on the single most common request in the protocol, since the header walk never looks at a body at all.
Pruning is a file operation or it is a rewrite. This node already prunes
bodies and receipts while keeping headers (docs/block-pruning.md) — a node
that serves history but has dropped bodies is a supported configuration.
Separate files make that "truncate the body file"; interleaved it means
rewriting every record to remove the middle of each one.
Body size is data-dependent and header locality should not be. Bodies range from near-empty to a few kilobytes with transaction count. Interleaved, how many headers land in a page depends on how busy the chain was that day, so the amplification varies with history rather than being a property of the format.
The two are read in different shapes. Headers come in runs of 192; bodies one at a time (§5). They want different record layouts, different readahead, and plausibly different block sizes. One file cannot be tuned for both.
Fetching a whole block — eth_getBlockByNumber with full transactions, or
replay — gets header and body in one read instead of two. That saves one seek
per block, on a path that handles one block at a time. Against 2.3× on the
hottest path in the protocol, it is not close.
Not in client sync time, on a real network. 10 GB of headers at 20 MB/s is 160 Mbit sustained, which is more than most peers will give. A real sync becomes network-bound long before it becomes disk-bound, so the client-side gain is largely theoretical.
In server capacity. Going from 37 MB/s to ~6 MB/s of disk per syncing client decides how many clients one node can serve at once. At today's rate a server saturates its disk on a handful of simultaneous syncs; with a freezer the same disk serves roughly six times as many.
That is the argument for #166: not faster syncs, but a network where serving them is cheap enough that nodes do it.
Both were read from source rather than documentation — /mnt/import/geth-src
and /mnt/import/reth-src. Several published descriptions turned out to be
wrong or silent on the points that matter most, and two of the answers below
contradict what the docs say.
Tables (core/rawdb/ancient_scheme.go). Five, each its own file series —
headers, hashes (canonical), bodies, receipts, bals (EIP-7928). Every
one snappy-compressed except hashes, which is stored raw.
Separate per kind, not interleaved. geth reached the same conclusion §5a reaches here, independently.
Index (core/rawdb/freezer_table.go:54):
type indexEntry struct {
filenum uint32 // stored as uint16 ( 2 bytes )
offset uint32 // stored as uint32 ( 4 bytes )
}
const indexEntrySize = 6Six fixed bytes per item. Locating block N is N * 6 into the .cidx file —
a multiply and a read, no search. Data files (.cdat) cap at 2 GB
(freezerTableSize), which is what the 2-byte file number buys.
Threshold — a formula, not a constant (core/rawdb/chain_freezer.go:132):
// max(finality, HEAD-params.FullImmutabilityThreshold)with FullImmutabilityThreshold = 90000. Under proof of stake finality is
normally well ahead of head−90,000, so the 90,000 is a floor for when finality
is unavailable, not the usual case — the opposite of what the documentation
implies.
Frozen data is deleted, not duplicated — and the ordering is the safety
argument (chain_freezer.go:218):
// Batch of blocks have been frozen, flush them before wiping from key-value store
if err := f.SyncAncient(); err != nil { log.Crit(...) }
// Wipe out all data from the active database
... DeleteBlockWithoutNumber(batch, ...); DeleteCanonicalHash(batch, ...)Sync the flat files first, delete from the key-value store second. A crash between the two costs duplicated data, never lost data. Genesis is always kept in the active database, and dangling side chains are deleted separately.
Not purely append-only. truncateHead and truncateTail both exist
(freezer_table.go:607, :718) — the first for reorgs reaching into frozen
range, the second for pruning from below.
Segments: Headers (carrying canonical hashes and terminal difficulties),
Transactions, Receipts, TransactionSenders, AccountChangeSets. Again
separate per kind.
DEFAULT_BLOCKS_PER_STATIC_FILE = 500_000 (crates/static-file/types/src/lib.rs:35).
Published summaries say 8,192; the source says 500,000.
Trigger: the static-file producer works from last_finalized_block per
segment rather than a fixed age — the same idea as geth's formula, expressed
differently.
Format: NippyJar — columnar, memory-mapped, with separate data, offsets and config files, compressed with zstd (optionally dictionary-based) or lz4. More machinery than geth's fixed six bytes, and more flexibility than headers need.
Fixed-width index, N * 6. The access pattern that matters is "give me the
192 headers below N", which is one index read and one contiguous data read.
NippyJar's columnar layout and dictionary compression buy flexibility this does
not need, at the cost of a format nobody here has debugged.
One file series per kind, for the reasons in §6 and because both implementations agree.
Delete after sync, never duplicate. geth's ordering exactly: write the flat files, fsync them, then delete from RocksDB. The failure mode is duplicated data, which is recoverable and detectable; the alternative failure mode is a hole in the chain.
The threshold cannot key off finality, because RSK has none. Both geth and reth freeze to the finalized block. There is no finality gadget here, so the question is what depth is deep enough that freezing is effectively permanent.
Existing constants to anchor to rather than inventing a number:
| constant | value | what it means |
|---|---|---|
MAX_REORG_DEPTH |
1,000 | deepest reorg the canonical index will follow |
MIN_KEEP_DEPTH (pruner) |
8,000 | the floor block pruning refuses to go below |
snapshot checkpoint_distance |
10,000 | how far back the state offered to peers sits |
Proposal: freeze below head − 8,000, reusing the pruner's floor. It is
already the number this node treats as "old enough to touch", it is eight times
the deepest reorg the index will follow, and reusing it means one concept to
reason about instead of two. geth's 90,000 would be ~31 days on RSK — far more
conservative than our own reorg behaviour justifies.
truncateHead still has to exist for the case the threshold is wrong, but at
8,000 it should never run.
- Headers first. They are the hot path — 192 per request, §4 — and the whole measured benefit is there.
- Receipts second, if
eth_getLogsturns out to be read-heavy enough to care. Not measured yet; should be before it is built. - Bodies last, because §5 says the protocol caps the win until body requests are batched.
Do not build the general "ancient store" of #166 first. The measured problem is header serving, the fix for it is one file series and one index, and the rest of #166 can follow the number it produces. Building the general thing first means carrying receipts and bodies through a design decision that headers alone would have settled.
Re-run the header phase of #184 against a freezer-backed server, on the same machine, same disks, same loopback, and record:
- throughput, against the 5,600–7,100 KB/s measured here;
- server disk read per byte delivered, against 6.5×;
- server and client CPU, to see which binds once the disk does not;
- random reads/s, which should collapse from ~5,300 to a few dozen.
Then the same for bodies, which needs an ordinary sync rather than a snapshot one, since a snapshot sync fetches only 6,000 bodies.
The prediction most likely to be wrong is §4's 20 MB/s ceiling, because it assumes CPU scales cleanly across two pinned cores and that nothing else appears once the disk stops binding. The one most likely to hold is §4's amplification figure, which is arithmetic over a known request size rather than a projection.
If §5 comes out at 5× rather than 2–3×, the likely reason is readahead doing more than expected, and that would be an argument for batched body requests rather than against them.