Skip to content

Developer Guide

Jump to bottom
Qubitium edited this page Oct 28, 2026 · 4 revisions

Developer Guide

Documentation home · Quickstart · Lifecycle

On this page

Overview

GPT-QModel separates model architecture descriptions, quantization algorithms, the layer-processing lifecycle, checkpoint formats, and inference kernels. A contribution usually touches more than one of these boundaries. An algorithm is not an inference backend; a checkpoint format is not a kernel; and adding a model to a registry does not prove its calibration or generation is correct.

This guide describes the source tree at revision c1976918cfe3703fbcf6d1ce2440a82b26394375, whose gptqmodel/version.py declares 7.7.0. Dependency ranges, support declarations, and CI infrastructure are revision-specific. Use the files in your checkout when contributing to another release.

Start with Quantization Lifecycle, Looper Processors, QuantizeConfig Reference, and Model Definitions and Module Tree. This page explains how to extend those systems and what must be validated before a change is ready.

Workflow at a glance

Contribution paths and contracts. Choose the extension boundary and validate the contracts it changes.

Reading the diagram: Lanes are independent contribution concerns; a change may span several. Calibration is method-dependent, not universal.

Text version of the workflow
  • Quantization method: Normalize configuration → Implement the processor → Pack and finalize → Validate the round trip.
  • Inference backend: Declare support → Implement packed compute → Check independent oracle → Audit persistent state.
  • Model architecture: Describe the module tree → Register the definition → Exercise the layer flow → Compare to the baseline.

Open full-size SVG

Concepts and ownership boundaries

Concept Source of truth Contributor responsibility
Public entry points gptqmodel/__init__.py, gptqmodel/models/auto.py Preserve load/dispatch behavior and public argument normalization.
Model architecture BaseQModel and gptqmodel/models/definitions/ Describe real module paths, execution subsets, shared inputs, modality handling, and model-specific requirements.
Quantization configuration gptqmodel/quantization/config.py Normalize configuration, reject invalid combinations, preserve serialization and reload semantics.
Quantization math gptqmodel/quantization/ Implement parameter search, calibration statistics, transformations, and quantized reconstruction.
Layer lifecycle gptqmodel/looper/ Capture inputs, execute processors, manage replay, pack/offload finalized modules, and support safe continuation.
Checkpoint layout FORMAT, packers, loader/writer Define tensor representation, scales/zeros/indexing semantics, and conversion boundaries.
Inference implementation gptqmodel/nn_modules/qlinear/ Declare the exact supported contracts and consume packed quantized weights correctly.
Native operator gptqmodel_ext/, gptqmodel/utils/cpp.py Build/load operators with the correct toolchain, ABI, architecture flags, and error/fallback behavior.
Tokenizer normalization External Tokenicer dependency Fix reusable tokenizer/chat-template behavior upstream, not in model-loader workarounds.

Repository layout

Python packages

Path Contents and when to edit it
gptqmodel/__init__.py Public package exports and initialization.
gptqmodel/models/auto.py GPTQModel, MODEL_MAP, config/model-type routing, backend/profile normalization, public loading entry points.
gptqmodel/models/base.py BaseQModel, model-tree setup, calibration preparation, quantize(), processor-chain construction, inference-facing model behavior.
gptqmodel/models/loader.py Model loading, checkpoint-backed/lazy materialization, model and tokenizer setup.
gptqmodel/models/writer.py Quantized model export, metadata, shard handling, optional quantization CSV output.
gptqmodel/models/_const.py Device/platform constants and availability detection.
gptqmodel/models/definitions/ Architecture-specific BaseQModel subclasses. Use a structurally similar definition, not merely a similarly named model.
gptqmodel/quantization/config.py METHOD, FORMAT, concrete config dataclasses, nested configuration, normalization, dynamic overrides, serialization.
gptqmodel/quantization/quantizer.py Scalar/group quantizer primitives, parameter finding, MSE search, Quantizer, QQQQuantizer.
gptqmodel/quantization/gptq.py, gptaq.py, foem.py, qqq.py Algorithm implementations and associated calibration/statistics work.
gptqmodel/quantization/awq/ AWQ search, scaling, clipping, and supporting modules.
gptqmodel/quantization/paroquant/, rotation/ ParoQuant optimization and rotation implementations.
gptqmodel/looper/ Processors, layer/subset execution, capture, execution configuration, checkpoint adapters/store/continuation.
gptqmodel/nn_modules/qlinear/ Quantized linear base hierarchy, concrete backend classes, runtime-cache helpers, quantized representation/forward paths.
gptqmodel/nn_modules/triton_utils/ Triton dequantization and GEMM support.
gptqmodel/adapter/, gptqmodel/eora/ Adapter abstractions and EoRA recovery functionality.
gptqmodel/exllamav3/ ExLlama v3 integration and native extension bridge.
gptqmodel/hf_kernels/, gptqmodel/hf_minimax_m2/ Bundled integration/support modules for particular model implementations.
gptqmodel/utils/ Kernel importing/selection, backend enums, JIT, model/Hugging Face utilities, concurrency, devices, logging, packing helpers, and engine bridges.
gptqmodel/extension.py Named extension discovery, loading, availability/errors, operator access, and cache clearing.

Native sources, tools, and documentation

Path Purpose
gptqmodel_ext/awq/ AWQ CUDA bindings and GEMM/GEMV sources.
gptqmodel_ext/exllamav2/, exllamav3/ ExLlama native implementations and bindings.
gptqmodel_ext/marlin/ GPTQ/AWQ repacking, FP16/BF16 Marlin GEMM sources, generated kernel support.
gptqmodel_ext/machete/, cutlass_extensions/ CUTLASS-based Machete implementation, code generation, shared native helpers. Machete's Readme.md documents its prepacking and code-generation model; it is not the public GPT-QModel loading API.
gptqmodel_ext/swordfish/, qqq/, vptq/, paroquant/ Native operators for the corresponding kernel/method families.
gptqmodel_ext/gptq_block/ GPTQ block quantization operators.
gptqmodel_ext/hadamard/ Hadamard-transform operators.
gptqmodel_ext/int4_qzero_group_scaled_cpu/ CPU packed INT4 operations, AVX512 support, bindings, packing helpers, and provenance notice.
gptqmodel_ext/floatx_cpu.cpp CPU low-precision conversion support.
hw/ascend_npu.md Ascend CUDA-to-NPU API equivalents and unsupported APIs. Do not assume CUDA capability/telemetry APIs exist unchanged on NPU.
optimize/_common.py CLI environment/thread setup, GPU-list validation, idle gate, visibility verification, calibration helpers, shared CLI arguments.
optimize/requant_embed_lm_head.py CLI for requantizing a checkpoint's input embedding, output embedding, or both.
scripts/ Reproduction, profiling, evaluation, and focused performance tools. Examples include benchmark_gptq_hessian_memory.py, benchmark_marlin_scratch.py, benchmark_gguf_cpp_vs_torch.py, profile_lazy_turtle_lifecycle.py, and dequantize_model.py. These are not a uniform installed CLI; inspect each script's arguments and fixture requirements.
tests/ Unit, numerical, integration, model, checkpoint, configuration, protocol, and benchmark coverage.
format/ Ruff configuration and the CI lint script.
.github/workflows/, .github/scripts/, .github/actions/ Workflow definitions, test planning/running/GPU allocation, and shared release actions.
checkpoint.md Operational contract for interruption-safe quantization and recovery.
docs/changelog/archive.md Historical release highlights. The archive explicitly covers releases before 7.3.4; do not derive current behavior from old release notes.
CREDITS.md, licenses/, native LICENSE/NOTICE files Attribution and upstream license/provenance information. Retain applicable notices when adapting code.
pyproject.toml, requirements.txt, setup.py, MANIFEST.in Build metadata, dependency requirements/extras, Python package discovery, native-source inclusion.

setup.py uses setuptools package discovery, including the gptqmodel_ext.* namespace packages. Native inference operators are managed by runtime/JIT loaders rather than a single eager build of all CUDA kernels during every package installation. Ensure new native sources are included in source distributions and are resolved from installed-package locations, not only a developer checkout.

Set up a development environment

The package declares Python >=3.10, setuptools >=84.2.1,<85, and dynamic dependencies from requirements.txt. At this revision those include torch>=2.14.1, transformers>=5.20.2, tokenicer>=0.0.14, and ninja>=1.13.2. These are the checked-in requirements, not a claim that an older release has the same minimums.

Install an appropriate PyTorch build for the target platform, then install the checkout and development extras in an isolated environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[test,quality]'
python -m pip check
Extra Declared purpose/dependencies
test pytest>=9.1.1, pytest-timeout>=2.4.0, parameterized>=0.9.0.
quality ruff==0.14.2.
eval Evalution>=0.0.17; required by the task-evaluation helper and evaluation-backed model tests.
bitblas bitblas==0.1.0.post1.
hf optimum>=2.3.0.
triton triton>=3.8.0 for Python <3.15.
mlx mlx>=0.32.4; mlx_lm>=0.31.3 for Python <3.11, otherwise >=0.32.2. Use Apple Silicon for MLX runtime validation.
vllm, sglang External engines with their own PyTorch/Transformers dependency constraints; pyproject.toml explicitly recommends dedicated environments.

The test extra supplies the test runner, not every integration dependency. For Evalution-backed tests, include eval in the editable install, for example python -m pip install -e '.[test,quality,eval]'; add only the runtime extras needed for the path under test.

For CUDA JIT contributions, an installed CUDA-enabled PyTorch wheel is not a replacement for a working compiler/toolkit and required headers. The shared JIT flag helpers select C++20 for Torch versions at least 2.14 and C++17 for older versions; the compiler must support the selected standard. For ROCm, use the compatible HIP/PyTorch toolchain and inspect the source-specific support checks. A CUDA device declaration does not automatically imply HIP support. For example, the AWQ GEMM packing helper selects the portable Torch packer on ROCm, XPU, NPU, CPU, and weight-only paths because the native AWQ GEMM extension is CUDA-specific.

See Installation and Quickstart, MLX Apple Silicon, and Troubleshooting and FAQ for platform and toolchain diagnosis.

Mandatory contribution rules

Quantized persistent storage

Read AGENTS.md before implementing or reviewing an inference path. Between uses, every inference backend must retain weights in their quantized representation. This applies to all methods, devices, accelerated and fallback paths, eager/JIT/compiled execution, and optional paths.

Allowed persistent state includes packed quantized layouts, scales, zero points, codebooks, and indexing metadata. Prohibited persistent state includes full dense/dequantized FP16/BF16/FP32 weights and expanded representations such as INT4 codes stored as full INT8 weights for reuse.

Temporary dequantization for the consuming computation is permitted; prefer bounded tiles and release references after consumption, including asynchronous work. Moving expanded weights to CPU, setting persistent=False, detaching them, using an opt-in environment variable, or describing them as a cache does not satisfy the rule. Allocator-reserved free memory is not a live tensor cache; this rule does not require flushing the allocator after each forward.

An existing fast path is not evidence of compliance. Inspect the state actually retained by the path under review and remove violations in that implementation.

Independent Torch numerical oracle

For quantization math, packing, inference kernels, dispatch, or runtime storage changes, read .agents/skills/gptqmodel-torch-oracle-numerics/SKILL.md before editing. The independent reference must perform its own arithmetic; expected outputs must not reuse the changed kernel or its intermediates.

The validation plan must cover:

  • Packed codes/indices and the representation's zero/scale semantics, not just a final perplexity score.
  • FP16 and BF16 separately wherever supported; shape, dtype, group boundaries, tails, non-contiguous inputs, repeated forwards, and fallback paths relevant to the change.
  • Arithmetic error separately from low-precision output rounding. In particular, premature FP16/BF16 scale rounding is not equivalent to rounding the final output.
  • Persistent state after first and repeated forwards, including fallback and opt-in branches. Check representation and lifetime, not merely names or serialization flags.
  • Allowed packed-cache invalidation on tensor mutation/replacement, checkpoint loading, and device/dtype moves; the explicit-clear contract for tensors without version counters.
  • Retained checkpoint bytes, additional packed-cache bytes, transient dequantization storage, numerical results, and speed. An old dense-cache benchmark can be historical context, not an eligible production optimization.

For every MLX inference kernel, AGENTS.md explicitly requires independent FP16 and BF16 activation tests, preservation of the input activation dtype, and comparison with an independent Torch oracle. Follow the skill's acceptance limits and output-rounding guidance rather than inventing a universal tolerance.

Tokenizer ownership and baselines

Read .agents/skills/gptqmodel-tokenizer-normalization/SKILL.md for tokenizer initialization, normalization, special-token compatibility, prompt rendering, chat templates, or unexpectedly low evaluation/generation quality.

Reusable corrections belong in Tokenicer. Add upstream tests, increment its version, run its complete suite, and submit the upstream change. GPT-QModel should consume the corrected dependency, not accumulate model-loader patches to avoid fixing it.

Before calling poor output a quantization regression, establish a non-quantized baseline with the same task/settings. Compare raw text, rendered prompts, input IDs, attention masks, special-token behavior, tokenizer/template versions, and aggregate scores. Task-specific few-shot formatting belongs to the evaluator; reusable tokenizer and template normalization belongs to Tokenicer.

Add a quantization method

1. Define the algorithm and format contracts separately

METHOD in quantization/config.py currently contains:

Member Serialized value
GPTQ "gptq"
GGUF "gguf"
FP8 "fp8"
BITSANDBYTES "bitsandbytes"
QQQ "qqq"
AWQ "awq"
EXL3 "exl3"
PARO "paroquant"
VPTQ "vptq"

FORMAT has these separate serialized-layout identifiers:

Member Serialized value
GPTQ "gptq"
GPTQ_V2 "gptq_v2"
GPTQ_P "gptq_p"
GGUF "gguf"
FP8 "fp8"
BITSANDBYTES "bitsandbytes"
MARLIN "marlin"
BITBLAS "bitblas"
QQQ "qqq"
EXL3 "exl3"
VPTQ "vptq"
GEMM "gemm"
GEMV "gemv"
GEMV_FAST "gemv_fast"
LLM_AWQ "llm-awq"
PAROQUANT "paroquant"

Do not add a METHOD for every optimization switch. GPTAQ/FOEM are configured through the GPTQ config/processor path. RTN has a concrete config and weight-only lifecycle but is not an additional METHOD member. Conversely, a genuinely different on-disk representation needs explicit format handling; renaming metadata without changing pack/load semantics is not a conversion.

2. Add and normalize the config

Use the existing concrete config pattern (GPTQConfig, AWQConfig, QQQConfig, ParoQuantConfig, etc.). QuantizeConfig is a backward-compatible factory whose metaclass selects a concrete class. Review all of these integration points in config.py:

Integration point Required work
METHOD, FORMAT and support/export-format tables Add valid method/format relationships; do not advertise unsupported exports.
Concrete dataclass and __post_init__ Establish actual defaults, normalize nested settings, validate cross-field constraints, and call parent normalization.
_resolve_quantize_config_class and payload normalization Route public factory construction and saved payloads to the correct class.
allowed_quant_methods(), supported_export_formats() Define accepted families and exported layouts.
from_quant_config(), from_pretrained() Handle checkpoint metadata, compatibility synonyms, and format inference intentionally.
to_dict(), _update_output_payload(), _update_meta_payload() Round-trip required algorithm/runtime parameters; place diagnostic metadata appropriately.
Dynamic config handling Apply per-module overrides without mutating a shared global config; explicitly reject unsupported overrides.
Package exports Expose the intended public configuration API from the relevant __init__.py files.

The factory filters constructor payloads to target-class fields. The saved-config parser also logs and ignores unknown fields. A misspelled argument can therefore disappear rather than behave as requested. Test normalized attributes and serialized output; successful construction alone does not prove the new setting is wired up.

3. Implement quantization math

The existing scalar quantizer constructor is:

def __init__(self, qcfg: BaseQuantizeConfig, shape=1, name: str=None):

The existing GPTQ algorithm constructor is:

def __init__(self, module: nn.Module, qcfg: Optional[QuantizeConfig] = None, region_timer=None):

These are different roles. Quantizer finds parameters and maps values; GPTQ owns per-module calibration/statistics and the algorithm execution. New methods need not inherit GPTQ if their mathematics does not fit it. Use the method's own implementation and the looper contract rather than pretending every algorithm has a Hessian.

In GPTQ, GPTQProcessor.preprocess() clones the module config, chooses GPTQ/GPTAQ/FOEM, configures per-channel quantization, and stores a task by module name. Its capture hook adds calibration batches; processing produces quantized state; finalization consumes scales, zeros and group indices to create/pack the quantized module. Follow that separation where appropriate.

4. Add the looper processor

Subclass LoopProcessor and choose an ExecutionConfig deliberately:

ExecutionConfig field Default Meaning
require_fwd True Whether the processor needs forward replay.
fwd_replay_after_process True Whether post-process replay produces authoritative next-layer inputs.
fwd_all_modules_in_single_pass False Combine module replay rather than replaying per subset.
subset_forward_early_stop False Stop a subset forward after its final required hook.
enable_activation_capture False Enable capture-only modules/hooks, including :? entries.
capture_layer_forward_context False Retain the original pre-process layer input/output context even when post-process replay supplies the authoritative next-layer cache.

The current GPTQ processor explicitly enables subset early stop and post-process replay. AWQ and ParoQuant also need activation capture. Weight-only processing does not need calibration forwarding. Copying one processor's flags without its lifecycle requirements can silently change the activations received by later layers.

The main override points are:

Hook Responsibility
preprocess() Build per-module tasks and apply module-specific configuration.
is_skipped() Tell the executor whether a module has work.
pre_process_fwd_hook() Collect the required input/output statistics.
process() Execute the per-module algorithm and publish state for later processors/finalization.
receive_layer_forward_context() Consume original layer input/output snapshots if required.
begin_shared_input_capture(), end_shared_input_capture() Elect/propagate capture leaders for explicitly shared inputs when supported.
register_moe_root_capture_hook() Optional method-specific MoE-root capture.
submodule_finalize() Replace/pack/offload each processed module and release temporary state.
finalize() Release shared resources after the model loop.
verify_calibration_dataset(), name() Validate/reuse calibration and identify the processor in logs.

The base processing signature is:

def process(
        self,
        module: NamedModule,
        device: torch.device = None,
        subset: Optional[Dict[str, NamedModule]] = None,
        previous_subset: Optional[Dict[str, NamedModule]] = None,
        subset_index: Optional[int] = None,
        subset_total: Optional[int] = None,
):

submodule_finalize() and finalize() are invoked in reverse processor order. Do not destroy state before a following processor such as EoRA consumes it. Handle concurrent devices, stream synchronization, shared input caches, and cleanup without assuming single-threaded execution.

5. Wire model orchestration and exports

BaseQModel.quantize() in models/base.py chooses packing/export behavior and constructs the processor chain. It explicitly dispatches EXL3, QQQ, VPTQ, AWQ, and ParoQuant processors; GPTQ uses GPTQProcessor; weight-only paths use WeightOnlyProcessor. Optional native capture/EoRA stages alter the chain.

Update both algorithm selection and exported-layout selection. Add an inference representation capable of reloading the result, implement the writer/loader metadata, and test a fresh-process reload. FORMAT.MARLIN is rejected for new quantization; GPTQ checkpoints can use the Marlin inference backend without exporting the deprecated layout. GPTQ BitBLAS is an inference repack path, not a direct quantization export.

6. Treat checkpoint support as a separate feature

Do not claim interruption-safe support merely because ordinary quantization works. looper/gptq_checkpoint.py validates models/configurations and matches exact processor-type tuples. QuantizationCheckpointAdapter must encode/decode the method's continuation state, packed modules, and identity. A new processor chain without an adapter is rejected.

Add supported-configuration tests plus fail-closed tests for changed source/calibration/config/runtime/device/packing identity, interrupted publication, damaged generations, and attempted concurrent writers. See Troubleshooting and FAQ and checkpoint.md before changing this contract.

7. Method acceptance checklist

  • Config defaults/normalization, invalid combinations, dynamic behavior, and serialization round trips.
  • Independent numerical oracle for quantization and packing, including supported dtypes and group/tail boundaries.
  • Calibration capture/replay and per-layer state cleanup.
  • Quantize, save, reload, and actual inference with the exported format.
  • Device/platform/optional-dependency rejection paths.
  • Persistent quantized storage and packed-cache lifecycle tests.
  • EoRA/checkpoint integration only where implemented and tested.
  • Method, configuration, compatibility, and user-facing documentation.

Add an inference backend

Choose the correct base class

Base class in nn_modules/qlinear/__init__.py Intended contract
BaseQuantLinear Shared module lifecycle, support declarations, validation and runtime behavior.
GroupedQuantLinear Adds group size, activation ordering, and symmetry contracts.
PackedGroupedQuantLinear Adds packed grouped representation behavior.
GPTQQuantLinear GPTQ packed-weight semantics.
PackableQuantLinear GPTQ raw packing/dequantization implementation used by pack-capable kernels.
AWQuantLinear AWQ grouped packed representation.
WeightOnlyQuantLinear Weight-only representations with their own initialization contract.

A new backend should live in a non-underscore module under gptqmodel/nn_modules/qlinear/ and expose a concrete subclass with its own SUPPORTS_FORMATS. Do not subclass an unrelated format merely to obtain a forward() implementation.

Declare support, then register the backend identifier

Add a canonical method-specific value to BACKEND in utils/backend.py. Add legacy normalization only if compatibility actually requires it. External-engine backends (VLLM, SGLANG, MLX) also have model/engine routing outside qlinear selection; an enum entry alone is insufficient for a new engine.

Declaration Required meaning
SUPPORTS_BACKENDS Enum values selecting the class. Avoid multiple classes matching the same method/format/backend tuple.
SUPPORTS_METHODS Actual algorithm families whose saved tensors can be consumed.
SUPPORTS_FORMATS Dictionary of FORMAT -> integer priority; positive priorities enter automatic selection, non-positive priorities retain explicit support but opt out of AUTO.
SUPPORTS_BITS, SUPPORTS_GROUP_SIZE, SUPPORTS_DESC_ACT, SUPPORTS_SYM Exact supported quantization contracts.
SUPPORTS_DEVICES, SUPPORTS_PLATFORM Device families and OS support; validate actual device ordinals/capabilities where needed.
SUPPORTS_DTYPES, SUPPORTS_PACK_DTYPES Activation/compute and packed-storage dtypes, not interchangeable concepts.
Shape divisibility and SUPPORTS_AUTO_PADDING Actual supported tails/group boundaries/padding.
Shard/training/adapter declarations Only capabilities backed by an implementation and tests.
SUPPORTS_RAW_PACKING where applicable Set False for load/repack-only classes that must not be advertised as public raw packers.

Kernel registration is source-driven. utils/importer.py imports non-underscore qlinear modules, walks BaseQuantLinear subclasses, and builds AUTO_BACKEND_KERNEL_MAPPING and BACKEND_TO_METHOD_FORMAT_MAPPING from support declarations. It sorts priorities descending, with backend string values as the descending tie-breaker. It skips module imports raising ImportError or OSError; other import failures are not a supported optional-dependency mechanism.

get_kernel_for_backend() rejects both missing and ambiguous matches. validate_quant_linear() caches immutable validation status using device-aware contracts; selection checks device families/capabilities, dtypes, packing, dynamic configurations, and other requirements. Do not replace this with a new unrelated hard-coded backend table.

Packing and initialization

Implement the correct packed tensor layout, zero-point convention, scales, group indices, and runtime repacking. For GPTQ-compatible packers, the existing API is:

def pack(
        self,
        linear: nn.Module,
        scales: t.Tensor,
        zeros: t.Tensor,
        g_idx: t.Tensor,
        block_in: int = 8192,
        workers: int = 8,
):

PackableQuantLinear.pack() delegates to pack_block(), whose own workers default is 1. Its block packing preserves the original qzero logic. Do not assume a backend's runtime layout is identical to the checkpoint layout. Distinguish raw pack, conversion/repack on load, post_init(), and device/dtype transitions.

Implement forward() to preserve the promised output shape/dtype, bias/adapter semantics, and unsupported-input checks. A portable fallback must honor the same mathematical and persistent-storage contracts as the accelerated path.

Native/JIT integration

Use the shared TorchOpsJitExtension in utils/cpp.py for native operators where appropriate. Its constructor accepts operator namespace/names, lazy sources/include/flag callbacks, build-root controls, ABI policy, CUDA requirements, rebuild/verbose settings, and optional prebuilt-library handling.

Important implementation properties:

  • Common default optimization is O3; the global optimization override is validated centrally.
  • C++ ABI flags come from the installed Torch build; source/toolchain/ABI-aware fingerprints separate incompatible cached builds.
  • CUDA architecture handling can merge visible capabilities into manual architecture overrides and restore the original environment afterward.
  • ROCm gets HIP-specific flags/link/include handling rather than blindly reusing NVCC flags. FMA contraction flags must preserve their numerical intent on both toolchains.
  • Thread/process synchronization protects JIT cache mutation. Cross-process lock waiting defaults to max(600.0, 5.0 * compile_baseline_seconds) seconds for the extension, with a missing baseline treated as zero. GPTQMODEL_TORCH_OPS_LOCK_TIMEOUT overrides this adaptive wait. Stale Torch baton cleanup occurs while holding the cross-process lock; do not remove the OS lock file to bypass an active build.
  • load() records build/lock/runtime errors. A failed optional extension can produce fallback behavior, but only if its caller implements a valid fallback.
  • Register a new publicly manageable native extension in gptqmodel/extension.py if it should participate in named load/status/error/cache operations.

Keep JIT module import lightweight and optional dependencies optional. Test both the successful native path and unavailable/build-failure behavior. For generated sources, include generated-layout coverage and reproducible generation requirements; preserve upstream credits and licenses.

Backend tests

Add selection tests and actual numerical execution tests. Cover explicit selection, AUTO ordering, unsupported method/format combinations, pack=True versus repack-only loading, dynamic overrides, activation/packing dtypes, all target devices, mixed-device rejection, shape tails, and cache invalidation. Use tests/kernels/test_selection.py, test_qlinear_hierarchy.py, format-specific kernel suites, and test_w4a_cache_lifecycle.py as relevant starting points.

Do not publish a broad SUPPORTS_* declaration based on a single GPU or one aligned shape. See Kernel Backends Reference for the user-facing compatibility contract.

Add a model definition

Describe the actual module graph

Create a definition in gptqmodel/models/definitions/ extending BaseQModel or the closest structurally compatible definition. The current Llama definition includes this real module tree:

module_tree = [
    "model",
    "layers",
    "#",
    {
        "input_layernorm": ("input_layernorm:!:norm",),
        "self_attn": ("q_proj:0:in=x", "k_proj:0:in=x", "v_proj:0:in=x", "o_proj:1"),
        "post_attention_layernorm": ("post_attention_layernorm:!:norm",),
        "mlp": ("gate_proj:0:in=x", "up_proj:0:in=x", "down_proj:1"),
    }
]

The complete parser/annotation contract is documented in Model Definitions and Module Tree. Subset ordering controls calibration execution; shared-input annotations permit Hessian capture deduplication only when the modules actually consume the same input. Do not infer shared input from matching shapes alone.

LlamaQModel also specifies pre_lm_head_norm_module = "model.norm", a verified model-type set for shared inputs, and AWQ shape-dependent handling for self_attn.o_proj. GQA/MHA shape differences can affect whether AWQ scaling applies. Copying just the module-tree string misses those requirements.

Register and validate

  1. Import the definition in models/auto.py and map the model config's exact lowercase model_type in MODEL_MAP. SUPPORTED_MODELS is derived from that map.
  2. Inspect model class/config availability in the installed Transformers version. Some registrations are conditional on Transformers support; a registry entry cannot provide a class absent from the dependency.
  3. Identify language layers, norm/head paths, unsupported/bypassed modules, tied parameters, experts, modality-specific components, batch support, and processor requirements from the real model implementation.
  4. Handle remote-code and model-specific load requirements using existing loader hooks. Keep reusable tokenizer normalization in Tokenicer.
  5. Add structural tests under tests/module_tree/ and model integration coverage under tests/models/. For MoE/multimodal models, verify routed experts and representative modality calibration, not only dense text linears.
  6. Run a non-quantized baseline, quantization, save/reload, and generation/evaluation with identical prompt/tokenization settings.
  7. Update the model documentation and compatibility claims only for the combinations verified.

Unknown model types can fall back to BaseQModel with auto-detected trees; this is a convenience path, not proof that a new architecture has a reviewed definition or method-specific correctness.

Running tests

Test organization

Location Coverage
Top-level files matching tests/test_*.py (glob) Loading, serialization, quantization config/lifecycle, concurrency/devices, logging, timing, and checkpoint regressions.
tests/kernels/ Kernel selection, pack/layout/numerical behavior, backend/device-specific execution, performance-oriented tests.
tests/qcfg/ Focused configuration behavior, including activation, dynamic overrides, and zero-point settings.
tests/module_tree/ Module-tree parsing and architecture/annotation behavior.
tests/models/ Real-model compatibility, quantization/evaluation harnesses, AWQ and modality/model-specific cases.
tests/protocol/ Protocol-focused tests.
tests/benchmark/ Benchmark workloads; do not confuse these with a small dependency-free unit suite.
tests/tasks/ Task fixtures/scripts such as ARC and GSM8K; excluded from recursive pytest discovery by tests/pytest.ini.
tests/fixtures/ Local fixture data.

tests/conftest.py seeds Torch, Python random, and NumPy with 787 and adds the repository/test/helper paths for the suite's mixed import styles. Run from tests/ to match helper-import conventions:

cd tests
python -m pytest kernels/test_selection.py -x -q
python -m pytest qcfg/test_dynamic_config.py -x -q
python -m pytest test_checkpoint_store.py -x -q
python -m pytest kernels/test_gptq.py -x -q

Choose only relevant suites for iteration; kernel/model tests may require specific GPUs, optional packages, network access, cached models, or local fixtures. A skipped hardware test is not successful validation of that backend. Use a real supported accelerator for changed accelerated paths.

Registered pytest markers

tests/pytest.ini sets addopts=-s -v, log_cli=true, and these markers:

Marker Declared meaning
ci CPU-only CI regression coverage for DeviceThreadPool affinity behavior.
cuda Requires a CUDA device.
cpu CPU-specific code paths.
gpu GPU-specific code paths.
inference Inference workloads replicating models across devices.
model Real weights or model-backed integrations.
mps Apple MPS-specific code paths.
slow Model loading, quantization, or benchmark-heavy paths.
timeout Requires pytest-timeout; retained for downstream compatibility.

For example:

python -m pytest kernels/test_selection.py -m 'not slow' -x
python -m pytest --collect-only -q

Marker filtering is not CI file allocation. The CI planner separately recognizes a leading # GPU=-1 comment before the first import as the no-GPU file marker. @pytest.mark.cpu alone does not implement that planning rule.

CI classification and model tests

.github/scripts/ci_workflow.py discovers test_*.py, applies ignore/config/regex rules, and builds CPU, Torch, model, and MLX lists. Model compatibility detection checks files under models/ for evaluation harness calls. MLX/Apple paths are classified separately; IPEX/XPU paths are excluded from the ordinary CPU/Torch/model lists. Per-file settings also come from .github/scripts/test.yaml.

Use the current test harness rather than inventing a new isolated model-download convention. Check the test's own constants/settings before launching a large integration run. Distinguish missing fixtures, unavailable dependencies, unsupported hardware, collection failures, and numerical regressions in reports.

Formatting and lint

Setting Checked-in value
Ruff version 0.14.2
Selected rules C, E, F, I, W
Ignored rules C901, E501, E741, W605, E402, F403, F405, E722, E701
Nominal line length 119; E501 is explicitly not enforced.
Import spacing Two lines after imports; gptqmodel is first-party.
Formatter Double quotes, spaces, automatic line endings, magic trailing commas retained.

format/format.sh changes into format/, installs the pinned Ruff, and runs ruff check ... --fix --unsafe-fixes. Review its edits. It targets models, nn_modules, quantization, utils, package initialization, docs/eora, tests and setup; it does not include every package, notably gptqmodel/looper/.

For explicit, non-mutating validation of changed Python paths from the repository root:

python -m ruff check --config format/ruff.toml gptqmodel/looper/loop_processor.py
python -m ruff check --config format/ruff.toml path/to/changed_file.py

Replace the second path with actual changed files. Configuration of formatter options does not mean the CI script runs ruff format; its command is ruff check. Do not run unsafe autofixes across unrelated files as a side effect of a focused contribution.

GitHub Actions and release infrastructure

Workflow Trigger and role
code_quality.yml Pushes and pull requests; Ubuntu 24.04, Python 3.13, runs format/format.sh.
unit_tests.yml Repository/workflow dispatch; builds CPU/Torch/model/MLX test plans, coordinates runners/GPU allocation, accepts test regex, parallelism and fast/slow model-test mode.
unit_tests_reusable.yml Shared self-hosted execution, environment setup, source distribution build, test invocation, log checks, GPU release.
compatibility.yml Pushes changing pyproject.toml and manual dispatch; invokes setuptools compatibility workflow.
setuptools_compatibility_reusable.yml Enumerates setuptools versions in >=84.2.1,<85 and checks build compatibility.
release.yml Published releases, repository dispatch, manual dispatch; selects self-hosted or GitHub-hosted source release flow and upload settings.

Unit-test workflow environment settings include forced JIT rebuilds, a multi-architecture CUDA target list, single visible CUDA device by default, disabled LogBar animation, and explicit Python/Torch/CUDA environment selection. These CI pins are not the same as public dependency ranges, and the caller and reusable workflow do not even declare the same Torch version at this revision:

Source Declared Torch version Declared Python version
.github/workflows/unit_tests.yml 2.11.0 3.14t
.github/workflows/unit_tests_reusable.yml 2.12.0 3.14t
requirements.txt / pyproject.toml >=2.14.1 >=3.10

Inspect the executing job's environment activation scripts and actual installed versions before interpreting a CI run as a supported public dependency matrix. The caller's environment label is not proof of the reusable job's installed Torch version.

The default dispatched unit runner is self-hosted infrastructure; copying the workflow to a fork does not provision its private services, containers, caches, or GPUs. The ordinary unit workflow is not a blanket push/PR gate in this revision. A green lint check is neither a numerical test nor complete source-tree lint coverage.

Pitfalls and review checklist

Pitfall Required review
Enum added, but nothing loads Verify config factory routing, processor/export selection, class discovery, support maps, and loader/writer paths.
Broad support declarations Run negative validation tests and per-device capability checks, not just happy-path inference.
Runtime repack mistaken for raw packing Check pack=True, SUPPORTS_RAW_PACKING, format identity, and fresh-process reload.
Config option appears accepted but has no effect Assert normalized concrete config fields; factory/parser filtering may discard it.
Module tree copied from another architecture Verify exact paths, forward ordering, shared inputs, shape-dependent AWQ rules, experts and modalities.
Faster kernel with a persistent dense cache Reject it under AGENTS.md; opt-in and non-persistent-buffer labels are not exceptions.
Numerical tests use the changed packer for expected output Build a separate reference; correlated implementation errors invalidate the oracle.
FP32-only accuracy test Test each supported low-precision activation dtype independently and separate final rounding.
Missing models/dependencies reported as a pass Record what executed, what skipped, and the exact unavailable prerequisite.
Checkpoint code changes without recovery tests Exercise publication, identity, fallback and interruption behavior, not only ordinary save/load.
Tokenizer/model quality regression guessed from scores Establish dense baseline and exact rendered prompt/token ID comparisons first.
CI lint assumed exhaustive Run pinned Ruff directly on all changed Python paths, including looper code.

Before submitting, provide the intended behavior, affected method/format/backend/model contracts, reproduction, tests/hardware/dtypes, numerical and storage evidence where required, and any unsupported paths. Preserve credits/license notices and update user documentation for new public functionality.

Maintaining this wiki

The wiki is a separate Git repository: https://github.com/ModelCloud/GPTQModel.wiki.git. Work against the source revision declared on Home; update version-scoped claims and source links together when refreshing for a new release. Do not turn a support declaration into an unqualified hardware guarantee.

Workflow definitions live in .tools/diagrams.json. The SVG renderer uses only Python's standard library. Edit the definitions rather than hand-editing generated SVGs. Preserve the distinction between alternative paths, independent concerns, and nested scopes.

From the wiki checkout, regenerate visuals and page contents:

python3 .tools/render_diagrams.py
python3 .tools/update_navigation.py
python3 .tools/validate_wiki.py
git diff --check

Before publishing, check Markdown links and anchors, SVG XML and text bounds, examples, and source paths. View representative diagrams at normal page width and inspect the published images after pushing. The SVGs contain accessible titles/descriptions; every embedding includes alt text and a collapsible text alternative.

Source files

  • AGENTS.md
  • .agents/skills/gptqmodel-torch-oracle-numerics/SKILL.md
  • .agents/skills/gptqmodel-tokenizer-normalization/SKILL.md
  • gptqmodel/__init__.py, gptqmodel/version.py, gptqmodel/extension.py
  • gptqmodel/models/auto.py, base.py, loader.py, writer.py, _const.py
  • gptqmodel/models/definitions/llama.py, gptqmodel/models/definitions/
  • gptqmodel/quantization/config.py, quantizer.py, gptq.py, qqq.py, awq/, paroquant/
  • gptqmodel/looper/loop_processor.py, gptq_processor.py, gptq_checkpoint.py, checkpoint_store.py
  • gptqmodel/nn_modules/qlinear/__init__.py, torch.py, marlin.py, _runtime_cache.py
  • gptqmodel/utils/importer.py, backend.py, cpp.py, env.py, logger.py, hf.py
  • gptqmodel_ext/, gptqmodel_ext/machete/Readme.md, MANIFEST.in
  • hw/ascend_npu.md, optimize/_common.py, optimize/requant_embed_lm_head.py, scripts/
  • tests/conftest.py, tests/pytest.ini, tests/kernels/, tests/qcfg/, tests/module_tree/, tests/models/
  • format/format.sh, format/ruff.toml
  • .github/workflows/, .github/scripts/ci_workflow.py, ci_tests.py, test.yaml, .github/actions/
  • pyproject.toml, requirements.txt, setup.py, CREDITS.md, checkpoint.md, docs/changelog/archive.md

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Using a model

Platforms

Internals

Clone this wiki locally