Repository navigation
* Deflate: encoder ~1.4-2x faster at equal or better ratio, decoder faster than zlib-ng - #454
Merged
Merged
Conversation
…kDotNet and a quick table The Deflate speed work needs a yardstick that is not anecdotal. DeflateCorpus builds five 8 MB inputs from files every checkout has: this repository's C# sources (code), the .NET reference pack's XML docs (prose), the runtime's managed DLLs (binary), the code corpus deflated once (already compressed), and runs of one value between log lines (repetitive). DEFLATE_CORPUS_DIR freezes them, so a before/after comparison across two checkouts sees the same bytes. DeflateBenchmarks runs the building block against System.IO.Compression (zlib-ng) at zlib levels 1, 6 and 9, compress and decompress, both decoders inflating the same stream into the same 64 KB buffer. The platform codec appears only in this non-product project, as the yardstick. `dotnet run -c Release -- deflate-table` prints ratio and MB/s for the edit-measure loop without a full BenchmarkDotNet run.
…r-resident loops Compression ran at about 55% of zlib-ng's speed at levels 6 and 9. A profile put 58% of level 6 in the chain walk at ~4 cycles a probe, so the number of probes, not their cost, was the lever: three-byte chains hold every string sharing three bytes, most of which cannot beat a match of four. - Chains are keyed on four bytes (15-bit multiplicative hash). Matches of exactly three come from a side table holding the latest position of each three-byte hash, taken within 4096 bytes (lazy levels, zlib's TOO_FAR) or 1024 (greedy levels). Ratio is equal or better at every level and corpus. - Runs of one byte are linked into their chain a vector at a time (IndexOfAnyExcept + sequential Vector128 stores) instead of a dependent head read/write per byte; same table contents. - The hot loops keep scan state, table and symbol-buffer references in locals (ref structs), the match search is out of line, and all of them are AggressiveOptimization: tier-0 code with OSR otherwise ran the first megabytes of every short-lived process. - Symbols are one literal/length index plus distance; emission folds codes, extra bits and bases into per-block tables and is branch-free (a literal takes distance slot 0, whose entry writes nothing), spilling 8 bytes per symbol with one room check. - The one-shot Compress presizes its MemoryStream to half the input. Frozen 8 MB corpora, Release x64, same process, cycles/byte ours vs zlib-ng: code L1 8.9->8.6 (5.2) L6 32.3->22.3 (19.0) L9 71.4->45.8 (40.4) prose L1 4.5->3.3 (2.4) L6 14.2->10.6 (8.6) L9 23.8->16.7 (18.5) binary L1 17.7->16.9 (9.0) L6 44.7->31.9 (27.8) L9 107.9->51.0 (64.1) repetit L6 3.09->0.65 (0.55) L9 4.86->2.03 (4.20) already compressed: ~10% slower (27-28 -> 30-33) from the side-table probe. Ratio (bytes, before -> after): code L1 2350857->2285939, L6 1868456->1866945, L9 1855786->1855997 (+0.01%); prose L6 657626->655782; binary L6 3864543->3859883, L9 3856356->3854456; repetitive unchanged. Level 1 stays ~1.7x zlib-ng's cycles: its deflate_quick writes fixed-Huffman blocks 20-25% larger; matching that would cost the ratio this keeps. Digests: PinnedFast and PinnedDefault in DeflateEngineTests change on purpose (new parse); PinnedBest is unchanged. Output still does not depend on write splits (now tested at levels 1/3/4/6/9 on runs and three-byte data). Sourcing: rung 3, RFC 1951 for the format and section 4's hash-chain/lazy scheme; zlib's per-level limits stay (rung 1, constants). Four-byte chains with a three-byte side table is the arrangement of zlib-ng (zlib licence) and libdeflate (MIT), rung 1 for the idea only; no code was converted.
…faster than zlib-ng Inflate ran at 0.7-0.8x zlib-ng. Every symbol paid for truncation and bounds checks and for two more lookups (base and extra-bit tables) after the Huffman one. - Table entries carry value << 16, flags, code length and code+extra bit count, so consuming a symbol is one shift by the entry and a length or distance needs no further lookup; reserved symbols and holes decode to an invalid entry. Root tables 11 bits (literal/length), 8 (distance). - A fast loop runs wherever 16 bytes of read-ahead and a match plus 32 bytes of output are free: one branch-free refill per step leaves >= 56 bits, so up to three literals or one length+distance go without checks; the next entry is looked up before the match copy; copies go 16 bytes (distance >= 16) or 8 bytes at a time, memset for distance 1. - A careful loop with every check handles the edges and hands back as soon as the fast loop can resume. Malformed input is rejected by both alike. - Both loops are AggressiveOptimization: the fast loop is entered once per ~96 KB, too rarely for tiering, and stayed tier-0 for whole runs. - Dynamic tables reuse their arrays across blocks. Cycles/byte, same streams, before -> after (zlib-ng): code 2.6 -> 1.6 (2.1), prose 1.0 -> 0.71 (0.75), binary 4.2 -> 3.2 (3.9), repetitive 0.19 -> 0.14 (0.11); 1.2-1.4 GB/s on code. Stored data stays at memcpy speed. Tests: every level x five data shapes x 25 boundary sizes through both decoders, zlib-ng output at levels 1/2/6/9 read back, 15-bit codes forcing sub-tables, every input/output chunking, 2400 bit-flipped streams and every truncation rejected with InvalidData/EndOfStream, never a crash. Sourcing: rung 3, RFC 1951. The entry layout (base and extra count folded in, the low byte driving the shift) follows libdeflate's decode-table design (MIT, rung 1, idea only; no code converted).
Hawkynt
force-pushed
the
perf/deflate-fast-paths
branch
from
October 9, 2026 12:35
997ce60 to
06c7b6a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the Deflate speed gap left by #452 as far as clean, safe-ish managed C# allows without giving up ratio.
What changed
Compression.Benchmarks): five 8 MB corpora (code, prose, binary, already-compressed, repetitive), a BenchmarkDotNet suite against System.IO.Compression (zlib-ng) at levels 1/6/9, anddotnet run -c Release -- deflate-tablefor a quick table. The platform codec is used only there, as the yardstick.AggressiveOptimizationon the hot paths.Measured (frozen corpora, Release x64, cycles/byte, before → after, zlib-ng in brackets)
Ratio is equal or better at every level/corpus (largest change: code L9 +0.01%). Already-compressed input compresses ~10% slower (side-table probe). Level 1 stays ~1.7x zlib-ng's cycles because zlib-ng's
deflate_quickwrites fixed-Huffman output 20–25% larger; level 6 stays ~1.15–1.25x because zlib-ng uses its cheaperdeflate_medium(worse ratio) while ours runs classic lazy matching with zlib's limits.Pinned digests
PinnedFast/PinnedDefaultupdated deliberately;PinnedBestunchanged.Verification
DeflateFastPathTests: levels 1–9 × 5 shapes × 25 boundary sizes through ours and zlib-ng; zlib-ng output at levels 1/2/6/9 read by ours; sub-table codes; input/output chunking; write-split determinism; 2400 bit-flipped streams and every truncation rejected cleanly.C:as a remote host (environment).