Skip to content

Use the current SIMD target's width for the transposed KV cache - #1027

Closed
Magnushst wants to merge 305 commits into
google:mainfrom
Magnushst:fix/flash-attention-nf
Closed

Use the current SIMD target's width for the transposed KV cache#1027
Magnushst wants to merge 305 commits into
google:mainfrom
Magnushst:fix/flash-attention-nf

Conversation

@Magnushst

Copy link
Copy Markdown

FloatsPerVector() returned hwy::VectorBytes() / sizeof(float), which is the vector width of the target that Highway's own library dispatches to. The flash attention kernels instead use hn::Lanes(df) of the target that Gemma dispatches to, and Gemma compiles a different set of targets (GEMMA_DISABLED_TARGETS). When the two widths differ, the transposed K/V cache is reshaped and written with one tile width but read with another.

For example, TestAttention in flash_attention_test aborts on the EMU128 target (Assert b != T2{0} in hwy::DivCeil): libhwy has no EMU128 code, so hwy::VectorBytes() returns 1 and the reshape factor is 0. On Arm CPUs with 256-bit SVE, such as Graviton 3, libhwy can dispatch to SVE_256 (8 floats), whereas Gemma disables SVE_256 and uses NEON_BF16 (4 floats). I have not been able to test on such hardware.

This takes the width from hn::Lanes in the per-target callers, passes it to MaybeReshapeCache, and removes FloatsPerVector(). The tile width is now also a compile-time constant on fixed-width targets, so TransposeKVCacheRow, which runs for every token, layer and KV head, no longer makes an indirect call through Highway's dispatch table.

Testing

  • flash_attention_test, with each case in its own process, on AVX2 and EMU128: before, 19/20 pass and TestAttention/EMU128 aborts; after, 20/20 pass.
  • The full CMake build (all targets) succeeds.

Balazs Racz and others added 30 commits December 16, 2025 01:56
…argets.

Creates new cc_library targets for :attention, :tensor_stats and :activations. Eliminates cyclic dependencies between these libraries.

PiperOrigin-RevId: 845690136
PiperOrigin-RevId: 846030124
PiperOrigin-RevId: 846140314
PiperOrigin-RevId: 846214605
PiperOrigin-RevId: 846648337
PiperOrigin-RevId: 846663686
Compute the L1 error and Shannon SNR (higher is better).

PiperOrigin-RevId: 846832280
PiperOrigin-RevId: 853138600
PiperOrigin-RevId: 853321311
Adds a SetMaxSeqLen method to ModelConfig to handle updating both max_seq_len and global attention window sizes. The Gemma constructor now checks if the provided inference seq_len exceeds the model's max_seq_len and, if so, emits a warning and updates the config.

This prevents clipping context to the hard-coded maximum.

PiperOrigin-RevId: 853676074
PiperOrigin-RevId: 853694463
PiperOrigin-RevId: 854171429
PiperOrigin-RevId: 862680527
PiperOrigin-RevId: 865932055
PiperOrigin-RevId: 867617121
…needle in a haystack of data.

Replaced flag with attention_impl to control which attention to run.

PiperOrigin-RevId: 869694868
PiperOrigin-RevId: 871776998
PiperOrigin-RevId: 871792281
PiperOrigin-RevId: 872280443
PiperOrigin-RevId: 874097322
…h reduces memory requirements by 4x on longer context on gemma models.

It also supports better parallelism for small batch sizes / small models.
It also is able to utilize VDPBF16PS for nice 2x improvement on avx512

PiperOrigin-RevId: 874517319
ZacharyGarrett and others added 29 commits August 27, 2026 12:14
…SbsWriterImpl, allowing export of models quantized to Microscaling FP4.

PiperOrigin-RevId: 972434552
…ints into Highway .sbs models, handling QKV projection fusing, per-layer embeddings (PLE) gating/projection/norm, and skip scalings.

PiperOrigin-RevId: 972444130
PiperOrigin-RevId: 972497820
PiperOrigin-RevId: 972619142
Call ToPtr directly. Also avoid = {init list} in tile_attention_test.

PiperOrigin-RevId: 974524857
activated_expert_idx[num_activated_experts] without incrementing
num_activated_experts. If num_activated_experts reaches kMaxExperts,
subsequent writes overflow the stack buffer.

PiperOrigin-RevId: 975079108
- When --mxfp4 is supplied, pack Safetensors E2M1 expert weights and
  per-32 E8M0 scale bytes directly into MxFp4Stream via WriteMxFp4Direct,
  avoiding intermediate F32 dequantization and re-quantization error.
- When --lossless or --mxfp4 is specified, convert FP8 weights directly
  to BF16 instead of down-quantizing to SFP.
- Configure ModelConfig with Type::kMXFP4 when --mxfp4 is set.

PiperOrigin-RevId: 975587935
CompressAndQuantizeQueriesMatrixAccumulationInt8Impl in gemma_cpp
to support queries provided as BF16 in addition to float.

Also update AbsMaxOfSpan to support BF16 inputs by promoting lower
and upper vector halves to float in registers, computing absolute
values, and reducing to the maximum float.

PiperOrigin-RevId: 975615108
   clusters via ParallelFor over o_groups.
2. Direct strided matrix view (C_g) into mla_o_mid destination buffer,
   eliminating intermediate copy per group.
3. Demote att_out directly from float to BF16 into per-cluster mla_o_in
   buffers, switching mla_o_mid to BF16 to reduce memory footprint and
   bandwidth.

PiperOrigin-RevId: 975630365
PiperOrigin-RevId: 975815177
`TopK` fills a `std::vector<double>` with one packed entry per logit, but
starts from an empty vector. `logits.size()` is the vocabulary size, so for
Gemma 3's 256K vocabulary libstdc++ reallocates 19 times and copies 2.1 MB
before the buffer reaches its final size. This happens on every sampled token
whenever `--top_k` is greater than 1.

Reserve the full size up front, and hoist the `accept_token` test out of the
loop so that the common case (no constrained decoding) is a straight-line
pack-and-append. Peak transient memory also falls, because the final doubling
no longer holds the old and new buffers at the same time.

Sampling results are unchanged: the same entries are appended in the same
order, so the subsequent VQSelect/VQSort see identical input.

Measured on an i9-13905H (AVX2, GCC 14.2, -O3 -DNDEBUG), best of 30 calls,
median of three interleaved baseline/patched runs:

  vocab 256K, k=50   367 us -> 284 us  (-23%)
  vocab  32K, k=50  43.4 us -> 33.3 us (-23%)

Add `TestTopK`, which covers both the filtered and the unfiltered path.
   zone breakdowns when compiled with -DPROFILER_ENABLED=1.
2. Link //third_party/highway:profiler into attention_benchmark in BUILD.
3. Instrument DSpark speculative decoding stages (Gen.MTP.Draft,
   Gen.MTP.VerifyPass, Gen.MTP.CommitDSparkKV) in deepseek_spec.cc with
   PROFILER_ZONE markers.
4. Add GCPP_ZONE profiling instrumentation to DeepSeek V4 MLA attention
   (ComputeQKV, DotSoftmax, SumHeads) and dense FFW in deepseek.cc.
5. Add PrintResults() call under #if PROFILER_ENABLED to run_dsv4.cc.

PiperOrigin-RevId: 976362424
PiperOrigin-RevId: 977763909
PiperOrigin-RevId: 977807618
…aveUpper

operate in-lane within 128-bit blocks, leaving 128-bit lanes 1 and 2
transposed when combined. Recombine the lower and upper halves across 128-bit
lanes using ConcatLowerLower and ConcatUpperUpper to preserve contiguous token order.

Fix BF16 SV tile query unpacking for 64-byte vector registers.

PiperOrigin-RevId: 977853596
…signed-by-signed (uint8 * int8) matrix multiplication for SV accumulation, improving quantization precision. Add SIMD emulation fallback using SumOfMulQuadAccumulate and TableLookupBytes.

PiperOrigin-RevId: 977934533
…ng the

polynomial evaluation to reduce serial latency on SIMD execution units.

Original formulation evaluated:
  v2 = Mul(v, v);
  arg = Mul(v, MulAdd(kMul, v2, kSqrt2OverPi));
which incurs 3 serial multiplications (~12 cycles) before entering vector Tanh.

The optimized formulation evaluates:
  v2 = Mul(v, v);
  v_kMul = Mul(kMul, v);
  v_kSqrt = Mul(kSqrt2OverPi, v);
  arg = MulAdd(v_kMul, v2, v_kSqrt);
which executes both initial multiplications concurrently on independent FMA ports
in ~4 cycles, reducing the critical path to ~8 cycles (2 multiply steps).

Additionally exposes GeluCdf and FastGeluCdf to allow fused FFN callers to evaluate
gated activation kernels (e.g. Mul(c2, GeluCdf(d, c1))) directly without redundant
vector multiplications.

PiperOrigin-RevId: 979091718
`FloatsPerVector()` returned `hwy::VectorBytes() / sizeof(float)`, which
is the vector width of the target that Highway's own library dispatches
to. The flash attention kernels instead use `hn::Lanes(df)` of the target
that Gemma dispatches to, and Gemma compiles a different set of targets
(`GEMMA_DISABLED_TARGETS`). When the two widths differ, the transposed
K/V cache is reshaped and written with one tile width but read with
another.

For example, `TestAttention` in `flash_attention_test` aborts on the
EMU128 target: libhwy has no EMU128 code, so `hwy::VectorBytes()`
returns 1 and the reshape factor is 0. On Arm CPUs with 256-bit SVE,
such as Graviton 3, libhwy can dispatch to SVE_256 (8 floats), whereas
Gemma disables SVE_256 and uses NEON_BF16 (4 floats).

Instead, take the width from `hn::Lanes` in the per-target callers, pass
it to `MaybeReshapeCache`, and remove `FloatsPerVector()`. This also
makes the tile width a compile-time constant on fixed-width targets, so
`TransposeKVCacheRow`, which runs for every token, layer and KV head, no
longer makes an indirect call through Highway's dispatch table.
@Magnushst Magnushst closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.