[Qubitium]

Summary

Extends GGUF CPU quantization, dequantization, loading and inference to all 37 supported aliases. Adds a gguf optional extra with stable minimum versions gguf>=0.19.0 and llama-cpp-python>=0.3.36; retains bitsandbytes>=0.50.2.

What Changed

• Alias/config lifecycle (quantization/config.py): _GGUF_BITS_ALIAS_INFO now covers all 37 aliases — q4_1, q5_0, q5_1, q8_1, q2_k/q3_k/q4_k/q5_k/q6_k/q8_k (+ _s/_m/_l quality variants), all iq*, tq1_0/tq2_0, mxfp4, nvfp4, q1_0, q2_0, q1_0_g128. bits=3 now defaults to q3_k. Variant grammar extended (xxs/xs/s/m/nl, mxfp/nvfp versions) so public format tokens like q_k_s or mxfp4 round-trip correctly.
• Quantize dispatch (qlinear/gguf.py, qlinear/gguf_cpp.py): gguf lib → internal impls (new Q8_1/Q8_K) → ggml_quantize_chunk for remaining K/IQ types. The ggml path is guarded by an explicit allowlist because upstream has no quantizer for Q8_1/Q8_K and assert(false) aborts the process on them.
• Dequantize dispatch: internal impls (incl. new Q8_1, Q8_K, I8/I16/I32/I64/F64) → ggml dequantize_row_* → gguf lib. Independent dequantization comparisons use rtol=atol=1e-6; this is a floating-point tolerance, not a bit-identity claim.
• Optimized CPU inference (GGUFCppKernel): native ggml_mul_mat on the packed weight (ggml's own SIMD vec_dot kernels; on this 8-core AVX2 Xeon it is 3-25x faster than the torch fused path and 10-100x faster than dense dequant+matmul), autotuned per (rows, dtype, qtype) against the torch fused/dequant paths with GPTQMODEL_GGUF_CPP_NATIVE_MATMUL override. Q8_1/Q8_K are excluded — upstream exposes no F32-source vec_dot for them and segfaults; they run the torch path. Weights stay quantized between calls (no persistent dense cache).
• llama-cpp-python does not export GGML_TYPE_Q8_1/TQ1_0/TQ2_0 constants; enum ids are supplied by a documented fallback table.
• GGUFTorchLinear, GGUFCppKernel, GGUFCudaKernel SUPPORTS_BITS extended to [1,2,3,4,5,6,8].
• Review fixes: Q8_1 now uses upstream half-away-from-zero rounding and the integer-code sum; Q8_K uses the signed maximum and reciprocal scale convention. Complete packed bytes match independent Torch and native ggml references, including threshold neighbors and zero blocks.
• CPU correctness/safety: reconstruct in FP32 before casting to FP16/BF16; reject malformed native buffers and input widths; handle empty inputs; keep Q8_1/Q8_K on Torch even under a forced-native override; clear the CPU autotune cache through the shared API.
• Thread safety: serialize access to ggml's shared mutable CPU graph scratch and guard bridge/backend initialization. Native buffers are freed after each operation; no dense weight cache is introduced.

Tests

• New tests/test_gguf_all_formats.py — 700 collected cases over all 37 aliases. Covers FP32/FP16/BF16 native, dense and fused CPU paths; independent packed-weight and A8 activation oracles; repeated-forward persistent-state inspection; complete config serialization; Q8 boundary bytes; invalid buffers; empty/padded inputs; forced fallback; cache clearing; and concurrent native calls.
• Verified formats produce packed sizes matching their upstream type_size (e.g. Q8_K=292 B/256, IQ1_M=56 B/256, NVFP4=36 B/64).
• Fast regression tests added and run.
• Targeted tests, Ruff and Python syntax compilation passed.

$ cd tests && python -m pytest test_gguf_all_formats.py test_internal_gguf.py test_weight_only_config.py test_bitsandbytes.py -q --tb=short
746 passed, 24 skipped in 20.95s
$ ruff check --config format/ruff.toml gptqmodel/nn_modules/qlinear/gguf.py gptqmodel/nn_modules/qlinear/gguf_cpp.py tests/test_gguf_all_formats.py
All checks passed!
$ python -m compileall -q gptqmodel/nn_modules/qlinear/gguf.py gptqmodel/nn_modules/qlinear/gguf_cpp.py tests/test_gguf_all_formats.py

The 24 skips are 12 intentionally unsupported native Q8_1/Q8_K cases and 12 unavailable bitsandbytes CPU packed-extension cases; Q8_1/Q8_K fallback tests pass. Tests ran on Linux x86_64 CPU with Torch 2.14.1+cpu. No standalone static type-checker is configured in this repo; syntax compilation was run, not a claim of full static type coverage.

Full-model test_gguf_qlinear_llama.py requires /monster/data/model/Llama-3.2-1B-Instruct, absent here. CUDA, AVX512/AMX and full-model generation were not validated on this AVX2 host. An earlier broader check found the unrelated transformers DummyCache.get_max_length compatibility failure.

Kernel Accuracy Requirements

• Removed self-comparisons from dequantization tests. Independent references are the gguf package and Torch byte decoders for Q1_0/Q2_0/Q8_1/Q8_K, plus native dequantizers where exported.
• Q8_1/Q8_K boundary tests: 5 blocks per format, source casts FP32/FP16/BF16; exact ties and adjacent representable values, saturation endpoints, signed-max ties, signed zero, and non-power-of-two scales. Complete packed-byte mismatches: 0 against both Torch FP32 and ggml; no tie exceptions needed. Dequantization drift against Torch: 0.
• Kernel tests use 16 output features, four input rows and widths 2 * block_size - 3 (61–509), including padding and multiple packed blocks. FP32 arithmetic is independently reconstructed before the specified output cast. Native/fused FP16/BF16 comparisons use the same final output rounding; dense Torch uses weights cast to the activation dtype. rtol=atol=2e-3.

Maximum absolute error against the applicable packed-weight oracle with exactly Q8-representable activations:

CPU path	FP32 output	FP16 output	BF16 output
Native ggml	8.94e-7	9.77e-4	7.63e-6
Torch dense	0	0	0
Torch fused	4.17e-7	9.77e-4	0

Native ggml additionally quantizes activations to Q8 for its SIMD dot kernels. The random-input tests compare that arithmetic to an independent Torch A8 oracle at rtol=atol=2e-3, separately from activation quantization loss. Max absolute arithmetic error is 0.0001683 / 0.0008669 / 0.0003659 for FP32 / FP16 / BF16 inputs, measured before the output cast. On the FP32 random-input fixture, native-vs-weight-only Torch max absolute drift is 0.026065; it is not just accumulation noise and does not satisfy a strict 0.002 weight-only equivalence guarantee. Use GPTQMODEL_GGUF_CPP_NATIVE_MATMUL=0 when weight-only semantics are required.

Repeated forwards retain only the packed qweight buffer: out_features * padded_blocks * type_size bytes (544–9,344 bytes in the oracle matrix), with no additional Python-owned dense/expanded tensor state. Dense fallback reconstruction is temporary FP32 scratch (up to 32,768 bytes for these shapes), plus its output cast; fused reconstruction uses bounded block chunks. Native CPU graph buffers, including their packed-weight copy, are freed after each call.

Review Requirements

• Reviewed this PR.
• Checked that the code matches existing project structure, APIs, and conventions.
• Avoided unnecessary monkeypatching and used the project's normal extension points where possible.
• Minimized and compacted the impacted code surface.
• Considered and eliminated potential regressions.

Notes

• Earlier CPU matmul measurements from the initial implementation (not rerun for the review fixes): rows=1/64, 8 threads, AVX2 Xeon, ms/call. These are host/shape-specific, not a universal speed guarantee. Native ggml includes activation quantization; Torch paths are weight-only.

fmt	shape	rows	ggml	torch fused	torch dequant
Q4_0	4096x4096	1	0.94	6.58	58.08
Q4_K	4096x4096	64	3.70	31.86	70.52
Q8_0	4096x4096	1	1.50	4.67	57.27
IQ2_XXS	4096x4096	1	0.53	57.55	99.00
MXFP4	4096x4096	1	0.98	67.15	117.85

• Follow-up candidates (out of scope): GGUFCudaKernel coverage for the new formats (CUDA path unchanged in this PR), oneDNN bf16x3 is still wired into the fused torch path for AVX512 boxes.
• Q8_1/Q8_K are inference-side only even upstream (KV-cache/aux types); we still ship internal quantizers so files can be produced, but matmul stays on the torch path.

Link to Devin session: https://app.devin.ai/sessions/bca72424bc9b493e8f91ef329ab9ff9b
Open in Devin Desktop: https://app.devin.ai/desktop/session/bca72424bc9b493e8f91ef329ab9ff9b?variant=devin
Requested by: @Qubitium

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring