Add runtime-selected LtHash backend with AVX-512 Blake3 XOF kernel

[masih]

Giga's flatkv LtHash spends most of a block's hashing time in the Blake3 XOF that expands each serialized key/value into 2048 bytes, followed by the scalar MixIn/MixOut over 1024 uint16 limbs. Profiling put Blake3 compression at roughly 56% of hashChunk and the two mixes at another 31%. The 32 XOF output blocks of one hash are independent compressions of the same chaining value with different counters, which maps directly onto a 16-lane AVX-512 kernel without any cross-mutation batching, and the limb arithmetic is a plain wrapping add/sub over 32 lanes of uint16.

This change moves the expand/add/sub steps behind a small backend struct selected once at init. The default backend is the existing pooled zeebo/blake3 XOF plus scalar mixing and always builds. A second backend, compiled only under goexperiment.simd && amd64 and enabled at runtime only when archsimd.X86.AVX512() and AVX512VBMI2() report support, runs a generated, fully unrolled 16-lane Blake3 compression using simd/archsimd (VPSHRDD for the rotates, pre-broadcast input rows loaded as vectors to avoid the legacy-SSE cost of Broadcast*) and vectorised limb mixing. Inputs longer than one Blake3 chunk fall back to the default expand. Output is byte-identical to the reference; SEI_LTHASH_BACKEND=default pins the portable path. A new workflow builds and tests the package both with and without the experiment, benchmarks every backend the runner CPU can execute, and writes a benchstat -col /backend comparison to the step summary.

Locally on a Xeon 8559C (AVX-512 + VBMI2), benchstat over 4 runs: Expand 2.61 µs → 1.04 µs, MixIn 213 ns → 18 ns, HashKV 3.13 µs → 1.08 µs, hashChunk (1000 mutations) 5.40 ms → 2.37 ms. Differential tests compare every backend against zeebo/blake3 across block and chunk boundaries (1..5000 bytes), existing lthash tests pass unchanged under both builds with -race, and golangci-lint run is clean with and without GOEXPERIMENT=simd.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 11, 2026, 8:49 PM

[codecov]

Codecov Report

❌ Patch coverage is 84.61538% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.48%. Comparing base (ac460ac) to head (356d8e3).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
sei-db/state_db/sc/flatkv/lthash/backend.go	50.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4151      +/-   ##
==========================================
- Coverage   66.60%   65.48%   -1.13%     
==========================================
  Files        2196     2090     -106     
  Lines      169188   157791   -11397     
==========================================
- Hits       112692   103325    -9367     
+ Misses      56355    54325    -2030     
  Partials      141      141              

Flag	Coverage Δ
sei-db	74.50% <ø> (ø)
sei-db-state-db	?
sei-db-state-db-pr	88.86% <84.61%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ei-db/state_db/sc/flatkv/lthash/backend_default.go	100.00% <100.00%> (ø)
sei-db/state_db/sc/flatkv/lthash/backend_nosimd.go	100.00% <100.00%> (ø)
sei-db/state_db/sc/flatkv/lthash/lthash.go	85.07% <100.00%> (-4.51%)	⬇️
sei-db/state_db/sc/flatkv/lthash/backend.go	50.00% <50.00%> (ø)

... and 156 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

LtHash default vs SIMD (uci-default)

CPU: AMD EPYC 7R13 Processor

This runner CPU lacks AVX-512F + VBMI2, so only the default backend ran.

goos: linux
goarch: amd64
pkg: github.com/sei-protocol/sei-chain/sei-db/state_db/sc/flatkv/lthash
cpu: AMD EPYC 7R13 Processor
             │   default   │
             │   sec/op    │
Expand-16      2.778µ ± 0%
MixIn-16       344.4n ± 0%
MixOut-16      344.8n ± 0%
HashKV-16      3.133µ ± 1%
HashChunk-16   6.384m ± 0%
geomean        5.806µ

          │   default    │
          │     B/s      │
Expand-16   703.2Mi ± 0%

[github-actions]

LtHash default vs SIMD (ubuntu-latest)

CPU: AMD EPYC 9V45 96-Core Processor

HashChunk is the end-to-end per-block path; vs base is simd relative to default.

goos: linux
goarch: amd64
pkg: github.com/sei-protocol/sei-chain/sei-db/state_db/sc/flatkv/lthash
cpu: AMD EPYC 9V45 96-Core Processor                
            │   default    │                simd                │
            │    sec/op    │   sec/op     vs base               │
Expand-4      3524.0n ± 1%   720.4n ± 0%  -79.56% (p=0.000 n=8)
MixIn-4       208.85n ± 1%   11.59n ± 3%  -94.45% (p=0.000 n=8)
MixOut-4      209.90n ± 2%   11.87n ± 3%  -94.35% (p=0.000 n=8)
HashKV-4      3726.0n ± 0%   742.4n ± 1%  -80.08% (p=0.000 n=8)
HashChunk-4    7.575m ± 0%   1.576m ± 1%  -79.20% (p=0.000 n=8)
geomean        5.344µ        649.9n       -87.84%

         │   default    │                 simd                  │
         │     B/s      │      B/s       vs base                │
Expand-4   554.3Mi ± 2%   2711.4Mi ± 0%  +389.18% (p=0.000 n=8)