Use the current SIMD target's width for the transposed KV cache - #1027
Closed
Magnushst wants to merge 305 commits into
Closed
Use the current SIMD target's width for the transposed KV cache#1027Magnushst wants to merge 305 commits into
Magnushst wants to merge 305 commits into
Conversation
PiperOrigin-RevId: 845153671
…argets. Creates new cc_library targets for :attention, :tensor_stats and :activations. Eliminates cyclic dependencies between these libraries. PiperOrigin-RevId: 845690136
PiperOrigin-RevId: 845883321
PiperOrigin-RevId: 846030124
PiperOrigin-RevId: 846140314
PiperOrigin-RevId: 846214605
PiperOrigin-RevId: 846648337
PiperOrigin-RevId: 846663686
…ode24 PiperOrigin-RevId: 846692015
Compute the L1 error and Shannon SNR (higher is better). PiperOrigin-RevId: 846832280
PiperOrigin-RevId: 853138600
PiperOrigin-RevId: 853321311
Adds a SetMaxSeqLen method to ModelConfig to handle updating both max_seq_len and global attention window sizes. The Gemma constructor now checks if the provided inference seq_len exceeds the model's max_seq_len and, if so, emits a warning and updates the config. This prevents clipping context to the hard-coded maximum. PiperOrigin-RevId: 853676074
PiperOrigin-RevId: 853694463
PiperOrigin-RevId: 853864585
PiperOrigin-RevId: 854110884
PiperOrigin-RevId: 854171429
PiperOrigin-RevId: 862680527
PiperOrigin-RevId: 865905454
PiperOrigin-RevId: 865932055
PiperOrigin-RevId: 867582875
PiperOrigin-RevId: 867617121
…needle in a haystack of data. Replaced flag with attention_impl to control which attention to run. PiperOrigin-RevId: 869694868
PiperOrigin-RevId: 871776998
PiperOrigin-RevId: 871792281
PiperOrigin-RevId: 872280443
PiperOrigin-RevId: 874097322
…h reduces memory requirements by 4x on longer context on gemma models. It also supports better parallelism for small batch sizes / small models. It also is able to utilize VDPBF16PS for nice 2x improvement on avx512 PiperOrigin-RevId: 874517319
PiperOrigin-RevId: 972087199
…SbsWriterImpl, allowing export of models quantized to Microscaling FP4. PiperOrigin-RevId: 972434552
…ints into Highway .sbs models, handling QKV projection fusing, per-layer embeddings (PLE) gating/projection/norm, and skip scalings. PiperOrigin-RevId: 972444130
PiperOrigin-RevId: 972456946
PiperOrigin-RevId: 972497820
PiperOrigin-RevId: 972529744
PiperOrigin-RevId: 972619142
PiperOrigin-RevId: 973789400
…ngth PiperOrigin-RevId: 974054967
Call ToPtr directly. Also avoid = {init list} in tile_attention_test.
PiperOrigin-RevId: 974524857
activated_expert_idx[num_activated_experts] without incrementing num_activated_experts. If num_activated_experts reaches kMaxExperts, subsequent writes overflow the stack buffer. PiperOrigin-RevId: 975079108
- When --mxfp4 is supplied, pack Safetensors E2M1 expert weights and per-32 E8M0 scale bytes directly into MxFp4Stream via WriteMxFp4Direct, avoiding intermediate F32 dequantization and re-quantization error. - When --lossless or --mxfp4 is specified, convert FP8 weights directly to BF16 instead of down-quantizing to SFP. - Configure ModelConfig with Type::kMXFP4 when --mxfp4 is set. PiperOrigin-RevId: 975587935
CompressAndQuantizeQueriesMatrixAccumulationInt8Impl in gemma_cpp to support queries provided as BF16 in addition to float. Also update AbsMaxOfSpan to support BF16 inputs by promoting lower and upper vector halves to float in registers, computing absolute values, and reducing to the maximum float. PiperOrigin-RevId: 975615108
clusters via ParallelFor over o_groups. 2. Direct strided matrix view (C_g) into mla_o_mid destination buffer, eliminating intermediate copy per group. 3. Demote att_out directly from float to BF16 into per-cluster mla_o_in buffers, switching mla_o_mid to BF16 to reduce memory footprint and bandwidth. PiperOrigin-RevId: 975630365
PiperOrigin-RevId: 975815177
`TopK` fills a `std::vector<double>` with one packed entry per logit, but starts from an empty vector. `logits.size()` is the vocabulary size, so for Gemma 3's 256K vocabulary libstdc++ reallocates 19 times and copies 2.1 MB before the buffer reaches its final size. This happens on every sampled token whenever `--top_k` is greater than 1. Reserve the full size up front, and hoist the `accept_token` test out of the loop so that the common case (no constrained decoding) is a straight-line pack-and-append. Peak transient memory also falls, because the final doubling no longer holds the old and new buffers at the same time. Sampling results are unchanged: the same entries are appended in the same order, so the subsequent VQSelect/VQSort see identical input. Measured on an i9-13905H (AVX2, GCC 14.2, -O3 -DNDEBUG), best of 30 calls, median of three interleaved baseline/patched runs: vocab 256K, k=50 367 us -> 284 us (-23%) vocab 32K, k=50 43.4 us -> 33.3 us (-23%) Add `TestTopK`, which covers both the filtered and the unfiltered path.
PiperOrigin-RevId: 976343397
zone breakdowns when compiled with -DPROFILER_ENABLED=1. 2. Link //third_party/highway:profiler into attention_benchmark in BUILD. 3. Instrument DSpark speculative decoding stages (Gen.MTP.Draft, Gen.MTP.VerifyPass, Gen.MTP.CommitDSparkKV) in deepseek_spec.cc with PROFILER_ZONE markers. 4. Add GCPP_ZONE profiling instrumentation to DeepSeek V4 MLA attention (ComputeQKV, DotSoftmax, SumHeads) and dense FFW in deepseek.cc. 5. Add PrintResults() call under #if PROFILER_ENABLED to run_dsv4.cc. PiperOrigin-RevId: 976362424
PiperOrigin-RevId: 976404275
PiperOrigin-RevId: 977763909
PiperOrigin-RevId: 977807618
…aveUpper operate in-lane within 128-bit blocks, leaving 128-bit lanes 1 and 2 transposed when combined. Recombine the lower and upper halves across 128-bit lanes using ConcatLowerLower and ConcatUpperUpper to preserve contiguous token order. Fix BF16 SV tile query unpacking for 64-byte vector registers. PiperOrigin-RevId: 977853596
…signed-by-signed (uint8 * int8) matrix multiplication for SV accumulation, improving quantization precision. Add SIMD emulation fallback using SumOfMulQuadAccumulate and TableLookupBytes. PiperOrigin-RevId: 977934533
…ng the polynomial evaluation to reduce serial latency on SIMD execution units. Original formulation evaluated: v2 = Mul(v, v); arg = Mul(v, MulAdd(kMul, v2, kSqrt2OverPi)); which incurs 3 serial multiplications (~12 cycles) before entering vector Tanh. The optimized formulation evaluates: v2 = Mul(v, v); v_kMul = Mul(kMul, v); v_kSqrt = Mul(kSqrt2OverPi, v); arg = MulAdd(v_kMul, v2, v_kSqrt); which executes both initial multiplications concurrently on independent FMA ports in ~4 cycles, reducing the critical path to ~8 cycles (2 multiply steps). Additionally exposes GeluCdf and FastGeluCdf to allow fused FFN callers to evaluate gated activation kernels (e.g. Mul(c2, GeluCdf(d, c1))) directly without redundant vector multiplications. PiperOrigin-RevId: 979091718
`FloatsPerVector()` returned `hwy::VectorBytes() / sizeof(float)`, which is the vector width of the target that Highway's own library dispatches to. The flash attention kernels instead use `hn::Lanes(df)` of the target that Gemma dispatches to, and Gemma compiles a different set of targets (`GEMMA_DISABLED_TARGETS`). When the two widths differ, the transposed K/V cache is reshaped and written with one tile width but read with another. For example, `TestAttention` in `flash_attention_test` aborts on the EMU128 target: libhwy has no EMU128 code, so `hwy::VectorBytes()` returns 1 and the reshape factor is 0. On Arm CPUs with 256-bit SVE, such as Graviton 3, libhwy can dispatch to SVE_256 (8 floats), whereas Gemma disables SVE_256 and uses NEON_BF16 (4 floats). Instead, take the width from `hn::Lanes` in the per-target callers, pass it to `MaybeReshapeCache`, and remove `FloatsPerVector()`. This also makes the tile width a compile-time constant on fixed-width targets, so `TransposeKVCacheRow`, which runs for every token, layer and KV head, no longer makes an indirect call through Highway's dispatch table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FloatsPerVector()returnedhwy::VectorBytes() / sizeof(float), which is the vector width of the target that Highway's own library dispatches to. The flash attention kernels instead usehn::Lanes(df)of the target that Gemma dispatches to, and Gemma compiles a different set of targets (GEMMA_DISABLED_TARGETS). When the two widths differ, the transposed K/V cache is reshaped and written with one tile width but read with another.For example,
TestAttentioninflash_attention_testaborts on the EMU128 target (Assert b != T2{0}inhwy::DivCeil): libhwy has no EMU128 code, sohwy::VectorBytes()returns 1 and the reshape factor is 0. On Arm CPUs with 256-bit SVE, such as Graviton 3, libhwy can dispatch to SVE_256 (8 floats), whereas Gemma disables SVE_256 and uses NEON_BF16 (4 floats). I have not been able to test on such hardware.This takes the width from
hn::Lanesin the per-target callers, passes it toMaybeReshapeCache, and removesFloatsPerVector(). The tile width is now also a compile-time constant on fixed-width targets, soTransposeKVCacheRow, which runs for every token, layer and KV head, no longer makes an indirect call through Highway's dispatch table.Testing
flash_attention_test, with each case in its own process, on AVX2 and EMU128: before, 19/20 pass andTestAttention/EMU128aborts; after, 20/20 pass.