Skip to content

Quantization Lifecycle

Jump to bottom
Qubitium edited this page Oct 27, 2026 · 3 revisions

Quantization Lifecycle

Documentation home · Quickstart · Lifecycle

On this page

Overview

BaseQModel.quantize() converts a loaded source model into a model whose selected modules contain packed quantized weights. It is not one model-wide weight conversion: the calibration path captures the decoder's initial inputs, replays layers in model-defined calculation groups, collects algorithm-specific statistics, solves each selected module, replays the reconstructed weights, and finally packs and optionally offloads the replacements.

There are two distinct execution paths:

Path Selection Orchestrator Calibration forwards
Calibration-based quantize_config.uses_weight_only_lifecycle() returns False ModuleLooper Input capture, subset statistics, and post-process replay as required by the processors
Weight-only That method returns True WeightOnlyLooper with WeightOnlyProcessor None; conversion uses the source weights directly

This page follows the implementation at source revision c1976918cfe3703fbcf6d1ce2440a82b26394375. Internal hooks, checkpoint schemas, and supported combinations are revision-sensitive. The processor reference is Looper Processors; configuration fields are described more broadly in QuantizeConfig Reference.

Workflow at a glance

Two quantization lifecycles. The configuration selects calibration-based or weight-only processing.

Reading the diagram: The lanes are alternatives. A later save() exports the model; it is not a quantization resume checkpoint.

Text version of the workflow
  • Calibration-based: Prepare calibration → Capture first inputs → Quantize layer subsets → Finalize model.
  • Weight-only: Select conversion → Walk weight targets → Convert and pack → Finalize model.

Open full-size SVG

Concepts and ownership

Object Owns Does not mean
BaseQModel Model definition, source-weight materialization, config, tokenizer, model-specific preparation/replay hooks, selected packing kernel An individual quantization algorithm
ModuleLooper Layer/subset order, processor chain, forward executor, device assignment, finalizer scheduling, extension boundaries A global choice of the numerical algorithm independent of the config
LoopProcessor Per-algorithm tasks, calibration cache, hooks, results, logs, packing/finalization One instance per linear; an instance serves many modules
NamedModule Stable relative/full names, layer index, module dtype, shared per-module state, synchronized access and stream-transfer helpers A separate persistent copy of all model weights
InputCache Positional layer inputs and batch-aligned replay metadata A generation KV cache
SubsetPlan Resolved execution policy for a calculation group or chunk A checkpoint or a user configuration object
DeviceAssignmentState Continuation of the quantization scheduler's device cursor and assignments Automatic remapping onto another GPU topology
LoopExtensions Synchronous start/boundary callbacks A background event bus or a serialization format

Three different products must be distinguished:

  1. Reconstructed weights are floating-point tensors produced by a quantizer. They are temporarily assigned to a dense module so later calibration replay sees the quantization error.
  2. Packed modules are replacements created in submodule_finalize(). Their persistent representation is method/backend-specific: packed codes and scales, codebooks, EXL3 tensors, or other supported quantized layouts.
  3. Saved inference models are produced separately by model.save(...). Neither a temporary offload directory nor a resumable quantization checkpoint is an inference export directory.

Public entry point

The callable signature in gptqmodel/models/base.py is reproduced below, with source comments omitted:

def quantize(
    self,
    calibration: Optional[Union[List[Dict[str, Union[List[int], torch.LongTensor]]], List[str], List[int]]] = None,
    calibration_concat_size: Optional[int] = None,
    calibration_sort: Optional[str] = "desc",
    batch_size: int = 1,
    tokenizer: Optional[PreTrainedTokenizerBase] = None,
    backend: Optional[BACKEND] = BACKEND.AUTO,
    adapter: Adapter = None,
    adapter_calibration_dataset: Union[List[Dict[str, Union[List[int], torch.LongTensor]]], List[str], List[int]] = None,
    calibration_data_min_length: int = 10,
    calibration_concat_separator: Optional[str] = None,
    embed_quant_config: Optional[Union[QuantizeEmbedConfig, QuantizeEmbed]] = None,
    embed_quant_mode: Optional[QuantizeEmbed] = None,
    checkpoint: Optional["CheckpointConfig"] = None,
) -> Dict[str, List[Dict[str, str]]]:
Argument Default Lifecycle effect
calibration None Required on the calibration path; ignored with an informational message on the weight-only path
calibration_concat_size None Passed to dataset preparation as calibration_dataset_concat_size
calibration_sort "desc" Descending length order; helper also recognizes "asc" and "shuffle"; other values preserve prepared order
batch_size 1 Dataset batching and processor replay/search input; model definitions can force it to 1
tokenizer None A supplied PreTrainedTokenizerBase replaces the wrapper tokenizer through Tokenicer.load; other types are rejected
backend BACKEND.AUTO Normalized together with method, format, activation mode, and device before validating/selecting the packing kernel
adapter None Replaces quantize_config.adapter when supplied; a Lora requests EoRA processing
adapter_calibration_dataset None Optional separate EoRA calibration dataset; otherwise EoRA uses calibration
calibration_data_min_length 10 Forwarded to dataset preparation on the ordinary path (see the forwarding note below); filters short examples
calibration_concat_separator None Forwarded to preparation; the helper treats None as an empty separator when concatenating
embed_quant_config None Normalized embedding conversion request; can also accept a QuantizeEmbed value
embed_quant_mode None Compatibility alternative; cannot be supplied together with embed_quant_config
checkpoint None Must be a CheckpointConfig; activates additional support validation, store lease, signal handling, and boundary persistence

The return value is the looper's mapping from processor labels to log-row lists. The annotation describes string-valued rows, but actual rows can contain additional numeric/structured diagnostic values. Treat the return as quantization diagnostics, not as the packed model; the model is mutated in place.

Minimal calibration-based use

from gptqmodel import GPTQModel, QuantizeConfig

config = QuantizeConfig(
    bits=4,
    group_size=128,
    device="cuda:0",
    offload_to_disk=True,
    offload_to_disk_path="./quant-work/offload",
)
model = GPTQModel.load("MODEL_REPO_OR_LOCAL_PATH", config, dtype="bfloat16")

# Supply representative text from your own calibration corpus.
calibration = [
    "A representative calibration passage with enough tokens to pass length filtering.",
    "Another representative passage covering the language and tasks used by the model.",
]
quantization_log = model.quantize(calibration, batch_size=1, calibration_sort="desc")
model.save("./quantized-model")

The small list illustrates the API, not a calibration-quality recommendation. LoopProcessor warns when the supplied dataset has fewer than 256 items or the prepared average input length is below 256; these are warnings, not minimum-size validation gates.

End-to-end call graph

BaseQModel.quantize(...)
  normalize embedding arguments; validate config/checkpoint/model state
  normalize method/format/backend; validate/select qlinear kernel
  optional tokenizer replacement, adapter override, rotation
  quant_blas_backend()
    uses_weight_only_lifecycle() == False
      _quantize_with_calibration(...)
        construct processors (optional Native / ModulePreProcessor / primary / Eora)
        optional DEVICE_THREAD_POOL.no_auto_gc()
        optional checkpoint_session(...)
        ModuleLooper(...).loop(...)
          tf32_high_precision_guard -> _loop_impl (inference_mode)
          prepare/reuse processor calibration datasets
          StageInputsCapture.cache_inputs -> InputCache
          release raw datasets; optional base-module disk offload
          build LoopPlan -> extensions.start(LoopContext)
          run_layer_stage(...)
            pre_quantize / materialize / resolve NamedModules
            for each processor and calculation subset:
              preprocess -> skip filtering -> build SubsetPlan
              run_subset_stage(...)
                register capture hooks
                ForwardExecutor.run -> serial or parallel -> run_single
                validate coverage / choose fallback / prune when policy requires
                DEVICE_THREAD_POOL -> processor.process(...)
              optional post-process forward replay -> receive_layer_inputs
            reverse processor submodule_finalize -> optional disk offload
            extensions.publish(LoopStep, finalizer futures)
          wait device/stream pools and finalizer watcher threads
          reverse processor.finalize -> return processor logs
    uses_weight_only_lifecycle() == True
      _quantize_weight_only(...)
        WeightOnlyProcessor -> WeightOnlyLooper.loop(...)
        resolve/materialize modules -> quantize_module -> submodule_finalize
        publish boundaries -> finalize -> return logs
  after_quantize(); optional W4A runtime stream installation
  flush region timer; set up online rotation; report dynamic-cache statistics
  return logs

The graph separates the initial model-entry capture forward, subset capture forwards, and post-process replay forwards. Initial capture delegates to the definition's run_input_capture() and stops at the first decoder layer's pre-hook; it is not a full-model inference pass. The base hook calls the model (or generate() when INPUT_EMBEDDING_EXTRA_ARGS is truthy), while multimodal definitions may prepare embeddings and call the first language layer directly. These forwards serve different purposes and can have different device-placement and output-retention policies.

1. Entry validation and preparation

quantize() first normalizes the embedding arguments and requires a non-None BaseQuantizeConfig. It then checks the retired resume environment variable, validates a supplied checkpoint request, rejects ordinary re-quantization of an already-quantized model, resets optional region timing, and checks the quantization-method blacklist.

Backend validation happens before the looper:

Case Entry behavior
FORMAT.MARLIN as a quantization output format Rejected; use FORMAT.GPTQ, with inference kernel selection handled separately
AWQ GEMV_FAST / LLM_AWQ export pack_dtype is set to torch.int16
Automatic AWQ GEMM packing Uses awq_gemm_pack_backend(...), accounting for device, ROCm, and weight-only export; native AWQ GEMM is CUDA-only, so portable packing is important on other hosts
NPU AWQ with a non-GEMM format in the automatic selection branch Rejected with an explanation requiring FORMAT.GEMM
QQQ automatic selection Portable BACKEND.QQQ_TORCH on CPU/NPU; otherwise BACKEND.QQQ
VPTQ / ParoQuant automatic selection BACKEND.VPTQ_TORCH / BACKEND.PAROQUANT_CUDA
FP8 / BitsAndBytes automatic selection BACKEND.FP8_TORCH / BACKEND.BITSANDBYTES
EXL3 public quantize() Only BACKEND.AUTO or BACKEND.EXL3_EXLLAMA_V3; requires torch.cuda.is_available() and a cuda quantization device, including HIP through PyTorch's CUDA API
BitBLAS format Checks BITBLAS_AVAILABLE and fails with its installation hint when unavailable

For non-EXL3 methods, select_quant_linear(..., pack=True) validates the proposed combination and selects self.qlinear_kernel. A processor can still have a specialized finalization class, such as QQQ, VPTQ, ParoQuant, or EXL3; packing backend and numerical quantizer are related but not interchangeable concepts.

If rotation is requested, the entry implementation limits model definitions to Llama/Qwen2, unties word embeddings when necessary, fuses norms, and rotates the model. MPS rotation uses CPU because MPS lacks float64 support. The accepted rotation backends are GPTQ_TORCH, GPTQ_TRITON, GPTQ_W4AFP8, and GPTQ_W4A_NVFP4; AUTO resolves to GPTQ_TORCH in this branch. See Rotation, Smoothing and Norm Tuning.

Config normalization disables offload_to_disk when rotation is enabled because the rotation path is incompatible with disk offload. The general offload default below does not override that normalization; rotation is also unsupported by checkpointed quantization.

The wrapper warns about partial dynamic rules for module units that inference engines fuse. It runs quantization under quant_blas_backend(); on ROCm this scopes the quantization-specific BLAS override rather than changing the user's inference policy.

2. Calibration data ingestion

The preparation chain is:

LoopProcessor.__init__(calibration=...)
  prepare_dataset_func(...)
    BaseQModel.prepare_dataset(...)
      gptqmodel.utils.calibration.prepare_calibration_dataset(...)

BaseQModel.prepare_dataset accepts lists of token dictionaries, strings, lists of token-ID lists, and supported Hugging Face dataset/iterable-dataset types. The helper also normalizes supported chat-message samples. Prefer explicit dictionaries when supplying already-tokenized data:

calibration = [
    {"input_ids": token_ids, "attention_mask": [1] * len(token_ids)}
    for token_ids in tokenized_examples
]
model.quantize(calibration)

tokenized_examples must contain representative, valid token-ID sequences. The helper validates dimensions, row lengths, and attention-mask compatibility, supplies masks where needed, handles text/message tokenization through the configured tokenizer, and trims rows against the resolved model positional limit. A bare string is rejected as a dataset. Empty datasets are rejected.

Processing order matters:

  1. Normalize and tokenize samples; validate token/mask structure.
  2. Trim overlong rows where the helper resolves a maximum-position limit.
  3. Remove examples with input length less than or equal to calibration_data_min_length (helper default 10).
  4. Optionally concatenate into fixed-size calibration blocks, including the configured separator.
  5. Sort ascending/descending by length, shuffle, or preserve order.
  6. Batch/collate according to batch_size and the model's batch-quantization support.

Minimum-length forwarding: the processor-construction paths bind quantize()'s calibration_data_min_length into prepare_dataset_func with functools.partial, so the public argument reaches the preparation helper on the ordinary path. Model prepare_dataset overrides accept it through **kwargs, but whether a multimodal adapter applies the same length filter is adapter-specific — check the adapter's own preparation contract rather than assuming uniform filtering.

LoopProcessor computes num_batches and total_calibration_tokens from prepared batches; token counting uses attention masks when available. Batch indices are stored in thread-local state during forward replay so capture order can remain deterministic even with multiple forward workers. Each processor has a cache; processors that return False from verify_calibration_dataset(index) can reuse the preceding processor's cache. Raw prepared datasets are released once initial input capture is complete.

An iterable accepted by the preparation helper is not a promise that every unsized generator can be passed directly to quantize(): processor construction calls len(calibration) before preparation. Supply a sized input or the supported dataset abstraction.

calibration=None is not equivalent to passing an explicitly empty dataset. The base processor initializes missing calibration as [] with zero batches, while several concrete verify_calibration_dataset() implementations test only for None. Do not rely on those checks to reject all missing/empty inputs before capture: provide usable calibration for activation-based methods and check actual captured coverage.

3. ModuleLooper setup and initial input capture

The constructor installs the shared ForwardExecutor, device pools and module-device assignment state, MoE policies, optional subset callbacks, extension dispatcher, and tracking for outstanding finalizer watcher threads. Model definitions supply the layer path and ordered module-tree calculation groups; they are not inferred solely from module-class names.

loop() enters the TF32 high-precision guard and calls _loop_impl() under torch.inference_mode(). The implementation preserves the model's config.use_cache value and disables cache while quantizing. It resolves the decoder layers and their dotted names, prepares embedding/lm-head defaults, and calls StageInputsCapture.cache_inputs(...) for processors needing an initial calibration cache.

StageInputsCapture

The stage installs a forward pre-hook on the first decoder layer. For each calibration batch, the ordinary model forward reaches that hook, which saves the layer's positional input list, attention mask, position IDs, extra keyword arguments, and model-specific capture data. It raises the intentional stop-forward sentinel so the remainder of the model need not execute merely to obtain the first decoder input.

The first layer's checkpoint path is resolved in this order:

  1. Caller-supplied layer_names[0].
  2. A legacy full_name attribute.
  3. Discovery by identity in model.named_modules().

A disagreement between the first two is logged; the caller's dotted path wins. A class name may label progress but is not used as a checkpoint prefix. The resolved path is passed to shell_module_materialize(..., module_path=...), which is essential for lazy source models. If the first layer has a deferred floatx source, capture can leave it lazy because the pre-hook runs before its packed weights are consumed.

Capture normally stages the first layer on CPU and stores cached calibration inputs on CPU. calibration_data_device can instead select a device or "balanced", which distributes cache storage across available forward devices, applying compute_device_filter where applicable. Storage placement is distinct from the device used by a later layer forward or quantization task.

InputCache schema

@dataclass
class InputCache:
    layer_inputs: List[List[torch.Tensor]]
    layer_input_kwargs: List[Dict[str, torch.Tensor]]
    position_ids: List[torch.Tensor]
    attention_masks: List[torch.Tensor]
    src_inputs: Optional[List[List[torch.Tensor]]] = None

When src_inputs is None, __post_init__() creates a shallow list copy of layer_inputs, or an empty list if no inputs exist. It does not clone every tensor. module_kwargs() returns position_ids and attention_masks; the latter is the cache's plural field name, not necessarily a layer's forward keyword. The executor translates the relevant per-batch metadata into the actual call.

Processors access this data through a thread-safe cache wrapper. receive_layer_inputs(...) replaces the inputs that feed the next layer after authoritative output replay.

4. Layer planning and execution: stage_layer.py

After capture, the looper releases calibration datasets, optionally offloads base modules, prewarms ParoQuant when present, and builds layer-module lists. With true_sequential=False, calculation groups are flattened rather than preserving sequential subset dependencies. This changes which quantized predecessors later statistics observe; it is not merely a scheduling optimization.

The looper builds an immutable LoopPlan containing the actual steps (input embeddings, transformer layers, and eligible output-embedding/lm-head steps). extensions.start(LoopContext(...)) runs before planned steps and returns the starting execution cursor for continuation.

For each layer, run_layer_stage(...) performs the model-specific preparation/materialization and module conversion, resolves concrete quantizable modules, creates NamedModule wrappers, and iterates processors in chain order. Full dotted names allow dynamic configuration, source loading, result storage, replacement, and device assignment to refer to the same object consistently.

For each processor it:

  1. Runs preprocess() to create tasks or annotate state.
  2. Applies is_skipped() and model/dynamic eligibility.
  3. Resolves subsets and their device/forward plans.
  4. Calls run_subset_stage() for each subset or expert chunk.
  5. Replays outputs after processing when execution_config.fwd_replay_after_process requires it.
  6. Updates the processor input cache for the next layer.

All consumers that need a dense/reconstructed module run before its base quantizer packs it. Per-module finalization follows the processor chain in reverse, allowing EoRA to preserve its adapter before the quantizer restores the base reconstructed weights and replaces the dense module.

Embedding steps and lm_head are explicit special cases, not ordinary decoder indices. Eligibility includes supported module types and tied-weight handling. In particular, weight-only lm_head conversion rejects tied weights rather than silently changing aliases. See Model Definitions and Module Tree.

5. Subset planning, coverage, and processing: stage_subset.py

SubsetPlan records the resolved module set, subset position, forward mode, replay policy, batching/chunks, row-count expectations, MoE group information, device assignments, and restoration policy for temporary forward-device overrides. It is a regular dataclass, not a Python-enforced immutable object; planning establishes decisions before workers run.

Plan property Meaning
need_forward_outputs True when the pass executes forward and does not obtain authoritative outputs from replay-after-process
subset_forward_serial Whether forward_mode == "serial"
batching_enabled Whether the plan contains more than one module chunk
calibration_coverage_policy Policy controlling validation, fallback, pruning, and dynamic-exclusion recording

run_subset_stage() has separate branches for chunked MoE work, ordinary forward-plus-process work, and no-forward processing. The inner pass registers module hooks, optionally enables shared-input capture or MoE-root capture, and executes cached batches through ForwardExecutor. Hooks are removed after capture rather than being retained on the exported inference graph.

Coverage is algorithm/policy-dependent. A routed expert may receive few or no tokens. The code can permit the configured fallback, remove uncovered targets, and record dynamic exclusions when the resolved coverage policy calls for those actions; it does not unconditionally quantize every unobserved expert with valid Hessian statistics. has_captured_input_ids() is the processor's coverage probe even when the underlying stored data is activations rather than literal input IDs.

The default Fallback configuration uses FallbackStrategy.RTN and threshold "0.5%". Per-module fallback choices are passed into processors. Fallback is not universal: EXL3 explicitly checks that its Hessian exists and that it captured positive samples, and ParoQuant has its own missing-feature handling.

After capture/coverage resolution, the looper assigns/reconciles module and task devices and submits processor.process(...) to DEVICE_THREAD_POOL. The worker receives the subset mapping, previous processed subset, and subset indices where applicable. Results are collected in a deterministic order even though worker completion may be concurrent. Stage cleanup synchronizes devices or flushes allocator caches according to the selected policy.

6. Cached forward replay: forward_executor.py

ForwardExecutor.run(...) chooses serial execution when forced, when auto_forward_data_parallel=False, or when there is only one usable forward device. Otherwise it uses the parallel implementation to distribute cached batches across forward-device replicas.

For one cached batch, run_single(...):

  1. Sets the processor's thread-local batch index.
  2. Moves positional inputs, masks, position IDs, and nested kwargs to the execution device.
  3. Supplies use_cache=False and lets prepare_layer_replay_kwargs adapt the call to the model definition.
  4. Executes the layer/module and handles shared KV/reuse state required by the model.
  5. Extracts the primary output when output collection is requested.
  6. Clears active attention-mask and batch-index state on exit.

Not every subset forward needs to retain outputs. A statistics-capture pass may early-stop once its last necessary hook fires; authoritative next-layer outputs can instead come from the full replay after process(). Processors needing pristine layer IO, notably grouped ParoQuant optimization, request that context explicitly.

Forward data parallelism and quantization-module distribution are different mechanisms. Replicating a layer for forwarding can increase VRAM and startup cost; more visible GPUs do not guarantee a faster run.

Device pools and scheduler continuation

Configuration field Default at this revision Role
device Method/config-dependent Quantization execution device; backend validation uses the resolved value
compute_device_filter None Filters candidate compute devices; an empty result is not treated as a request to disable all computation
calibration_data_device None Initial calibration-cache storage; supports "balanced"
auto_forward_data_parallel True Permits cached-batch data parallel forwarding
dense_vram_strategy VramStrategy.EXCLUSIVE Dense/serial-path module placement
dense_vram_strategy_devices None Optional dense device pool
moe_vram_strategy VramStrategy.EXCLUSIVE Expert-family placement
moe_vram_strategy_devices None Optional expert device pool

For balanced MoE quantization, use logical indices relative to CUDA_VISIBLE_DEVICES. The dense and MoE pools are separate, and expert families remain together. Actual free-threaded parallel MoE eligibility also depends on the Python runtime: PYTHON_GIL=0 requires a free-threaded build and a genuinely disabled GIL. It cannot convert a regular build into one. ParoQuant is restricted to single-device quantization by the looper at this revision.

DeviceAssignmentState in execution_state.py is shared by the calibration and weight-only orchestrators. It serializes a version-1 mapping containing next_device and module_devices. Restore rejects a negative/non-integer cursor, invalid names, and devices outside the current quantization pool plus CPU. This state preserves future scheduling as well as already assigned modules.

For pool examples see Memory Offload and Multi-GPU.

7. Preprocessing, dense sources, and linear runtime modes

ModulePreProcessor is a no-forward LoopProcessor inserted when quantize_config.preprocessors is nonempty. Its preprocess() records an ordered preprocessor_pipeline and specialized state:

Preprocessor Recorded state Consumer
AutoModuleDecoderConfig auto_module_decoder, decode/passthrough policies, optional native floatx runtime plan Model materialization/source hand-off
TensorParallelPadderConfig tp_pad_info with pad_cols, target_multiple, and original_columns Quantizers and adapter/dequantization handling
SmootherConfig Serialized entry in the ordered pipeline Quantization/source preprocessing machinery

The TP target multiple is the least common multiple of (2, 4, 8) and a positive group_size when present. The planner annotates; its process() is a no-op. Source decoding is deferred until quantization hand-off rather than being repeated during post-quantization W4 replay. Processor-specific source lookup can use quant_source_module rather than treating every live forward wrapper as an ordinary dense source.

linear_mode.py defines:

class LinearMode(str, Enum):
    INFERENCE = "inference"
    TRAIN = "train"

This is a runtime mode for fused Torch linear implementations, not a selector between ModuleLooper and WeightOnlyLooper. In torch_fused.py and torch_fused_awq.py, an initially unset mode becomes INFERENCE only when the module is not training, the input does not require gradients, and TORCH_HAS_FUSED_OPS is true. Otherwise it becomes TRAIN, using the corresponding dequantize-and-matmul path. The branch checks the input's requires_grad, not the global torch.is_grad_enabled() flag, and is selected only while linear_mode is None. Do not infer lifecycle scheduling or checkpoint support from this enum.

8. Finalization, offload, and garbage collection

process() completing does not imply that packing or disk writes are complete. stage_layer.py queues per-module finalizers; these call submodule_finalize(), install packed replacements, and perform method-eligible offload. Finalizer exceptions request loop termination and must be surfaced before claiming a safe checkpoint.

Field Source default Effect
offload_to_disk True Enables quantization-time offload of base/completed modules
offload_to_disk_path None Auto-created temporary directory when disk offload is enabled and no path is supplied
gc_mode GcMode.INTERVAL Normal periodic thread-pool GC policy
wait_for_submodule_finalizers False True explicitly waits for per-layer finalization before the next layer; False permits overlap unless a forced-synchronous condition applies

The enum is:

class GcMode(str, Enum):
    INTERVAL = "interval"
    ON_STAGE_END = "on_stage_end"

ON_STAGE_END enters DEVICE_THREAD_POOL.no_auto_gc() at the model entry wrapper and requests explicit cache cleanup at stage boundaries. It is not a tensor offload mode and does not make live tensors disappear. Disk offload lowers live tensor residency but the accelerator allocator may still retain freed pools until cache cleanup.

Even with wait_for_submodule_finalizers=False, the layer stage drains finalizers synchronously when W4A activation modes are active, when more than one non-CPU quantization accelerator is active, or when ParoQuant participates. Otherwise it may use a SubmoduleFinalizeWatcher thread so packing/offload overlaps the next layer. The looper tracks these threads and waits for them before overall completion. Certain synchronous finalization cases also release the accelerator cache to limit allocator growth.

For the GPTQ/QQQ/AWQ/ParoQuant paths, finalized modules are eligible for forced disk offload. This stage's processor-type guard does not include EXL3 or VPTQ; do not assume identical per-module offload behavior for every algorithm. A missing path in the guarded offload branch produces a warning and skips that write rather than inventing a destination. Ordinary config initialization normally supplies a path automatically.

The generated directory uses a gptqmodel_ prefix and is owned by the config's shared temporary-directory object. Set an explicit path when you need operational control over storage lifetime. Do not remove offload files while live model hooks, finalizers, checkpoint attempts, or saving still depend on them.

On successful calibration-loop completion, the looper waits for DEVICE_THREAD_POOL, dangling watcher threads, and the stream-device pool; collects processor logs; finalizes processors in reverse; and restores config.use_cache. The public entry then runs after_quantize(), installs the applicable W4A runtime stream, flushes timers, and sets up online rotation. model.save() remains a separate operation.

Boundary extension contract

Extensions are an internal orchestration interface. Both loopers accept extensions=() in their constructors; public quantize() has no general-purpose extensions argument. The checkpoint wrapper is one caller that supplies an extension.

API Contract
LoopStep(kind, index, name) Frozen execution identity; its index need not be a decoder-layer index
LoopPlan(steps) Requires an exact tuple of unique steps; after(step) returns the following cursor
LoopContext(plan, model, processors, shared_state, execution=None) Frozen context passed to start callbacks; execution implements pool identity and scheduler continuation
Optional extension.on_start(context) Install continuation before planned execution; return None or an integer cursor from 0 through the plan length
extension.on_boundary(boundary) Observe a completed step before the next step mutates execution state
boundary.quiesce() Wait for every preceding future, then return their results or propagate a worker failure
LoopExtensions.start(context) Validates returned cursors and rejects conflicting non-None cursors from different extensions
LoopExtensions.publish(step, futures=()) Dispatches callbacks synchronously and retains unfinished, failed, or cancelled futures for later quiescence

Boundaries are borrowed only during the callback and belong to the orchestration thread. Calling quiesce() later or from a worker raises RuntimeError. It waits for all futures before raising a failed future's exception so recovery cannot run while other finalizers are still mutating the model. Extensions must quiesce before treating a layer's outputs or packed/offloaded state as durable. extension.py itself knows nothing about tensors, storage formats, or checkpoint policy.

9. Checkpoints and continuation/resume

Supported scope is deliberately restricted

Checkpointing captures quantization execution, not an arbitrary Python process or an exported inference model. validate_checkpoint_support() requires:

Requirement Current restriction
Model type llama or qwen3_moe
Layer order true_sequential=True
Disk backing offload_to_disk=True
Source Local model.safetensors or model.safetensors.index.json plus referenced shards and config.json
Disallowed config features Dynamic rules/exclusions, GPTAQ, FOEM, rotation, lm_head quantization, and embedding quantization
Adapter No adapter or an exact Lora instance; EXL3 additionally rejects adapter/EoRA generation
Thread Quantization must run on the main thread for scoped signal handling
Placement serialization CPU and CUDA device tags; ROCm uses PyTorch's cuda namespace; no general MPS/XPU/NPU continuation adapter

QuantizationCheckpointAdapter.bind() separately permits these exact processor chains:

Chain Notes
(GPTQProcessor,) or (GPTQProcessor, EoraProcessor) Standard GPTQ, not GPTAQ/FOEM
(AWQProcessor,) or (AWQProcessor, EoraProcessor) Preserves scaling mutations as well as packed modules
(QQQProcessor,) or (QQQProcessor, EoraProcessor) Preserves QQQ packed schema
(ParoQuantProcessor,) or (ParoQuantProcessor, EoraProcessor) Preserves Paro rotation/runtime options
(WeightOnlyProcessor,) Shares the boundary/state-adapter protocol
(EXL3Processor,) Added when the configured method is EXL3

VPTQ, NativeProcessor, DequantizeProcessor, and chains containing ModulePreProcessor are not on this exact-chain allowlist. A config being valid for ordinary quantization is not evidence that it supports checkpointing.

Configuration and usage

@dataclass(frozen=True)
class CheckpointConfig:
    path: str | Path = "auto"
    resume: str = "auto"
    interval: str = "layer:1"
    keep_last: int = 2
    strict_device_check: bool = True

The class also keeps an internal, non-init _resolved_path field so an automatic root resolves once across retries.

Field Accepted values / behavior
path Nonempty string/path; "auto" resolves to the config's offload root. An explicit path is preferable for restarting with newly loaded config/model objects
resume "auto": restore if a generation exists; "required": fail if none exists; "never": require a directory without published checkpoints
interval "layer:<positive integer>"; whitespace around the full expression is stripped for validation
keep_last Integer at least 2; retains fallback generations
strict_device_check Boolean, default True; False removes only physical UUID/serial from identity comparison, not topology/capability/pool requirements 
from gptqmodel import CheckpointConfig, CheckpointStopped, GPTQModel, QuantizeConfig

# Use a supported llama or qwen3_moe source without the excluded features.
config = QuantizeConfig(
    bits=4,
    group_size=128,
    device="cuda:0",
    true_sequential=True,
    offload_to_disk=True,
    offload_to_disk_path="./quant-work/offload",
)
model = GPTQModel.load("LOCAL_SOURCE_MODEL", config, dtype="bfloat16")
checkpoint = CheckpointConfig(
    path="./quant-work/checkpoints",
    resume="auto",
    interval="layer:1",
    keep_last=2,
)
try:
    model.quantize(calibration_dataset, checkpoint=checkpoint)
except CheckpointStopped:
    # A cooperative SIGINT/SIGTERM stop committed a safe boundary.
    pass
else:
    model.save("./quantized-model")

Resume by loading the same local source model and supplying the same calibration/configuration and checkpoint root. The code still constructs the execution plan and initial capture needed to validate identity before installing continuation. A checkpoint is not permission to substitute a different calibration corpus or skip source availability.

Composition and publication order

File Responsibility
extension.py Immutable execution steps, start context, borrowed boundaries, finalizer quiescence
checkpoint.py CheckpointExtension: interval policy, restore cursor, cooperative stop, capture/commit orchestration
gptq_checkpoint.py Model/processor support validation, session lease/signal wrapper, identity and quantization-state adapter
checkpoint_store.py CheckpointConfig, writer lock, immutable objects, manifests, atomic CURRENT, validation/retention/collection
checkpoint_devices.py Visible-device identity and optional UUID/serial projection
checkpoint_modules.py Explicit packed-module schemas and constructors; no dynamic import driven by checkpoint data
continuation.py Safe tensor/value encoding and device-aware decoding
execution_state.py Shared scheduler continuation

checkpoint_session() takes an exclusive writer lease for the root, creates an attempt-private attempt-* offload directory, and redirects the model config's offload path there. It temporarily replaces SIGINT/SIGTERM handlers with a stop request and restores previous handlers on exit. The lower-level extension itself does not install global signal handlers. The lease encloses setup, execution, and worker cleanup so a failed attempt cannot race another writer.

At a due boundary, including the final planned step or a requested stop:

  1. LoopBoundary.quiesce() waits for all preceding finalizer futures and propagates failures.
  2. The adapter captures processor caches/logs/results/statistics, shared state, packed module specs/artifacts, materialized dense parameters/buffers and aliases, device scheduler state, and Torch/CUDA/Python/NumPy RNG states.
  3. ContinuationCodec creates one immutable safetensors object containing a JSON value tree and owned tensor bytes.
  4. The store writes content-addressed SHA-256 objects and a manifest referencing all required artifacts, including earlier completed layers.
  5. An fsynced atomic replacement of CURRENT publishes the new generation.
  6. The adapter updates committed artifact locations; unreachable objects are collected after publication.
  7. If stopping was requested, CheckpointStopped is raised only after the safe generation is committed.

An interrupted write is not recovered by scanning for the newest-looking file. Only generations named in CURRENT are candidates. The store tries the newest complete published generation and can fall back to retained complete generations when a newer one is corrupt. Identity mismatch fails closed; it is not treated as ordinary corruption to be silently bypassed. A post-publication cleanup failure logs a warning rather than turning a durable commit into a failed transaction.

Identity and topology

The adapter hashes actual source configuration/shards and initial processor caches, not merely model IDs or calibration filenames. Its identity includes algorithm configuration (including runtime/method-specific fields), processor chain, packed kernel, runtime Torch/Transformers versions, execution plan, and device topology. Offload-path/temporary-directory/telemetry fields are excluded from quantization identity so attempt-private storage can change without changing the numerical configuration.

Strict topology compares visible GPU ordering, UUIDs, serials when available, names, capabilities, selected pools, and device counts. If any execution pool uses CUDA, topology capture inspects every visible CUDA device, not only devices selected for work, and requires each device's PyTorch properties to expose a UUID. This also matters on ROCm, which uses the CUDA API: successful ordinary quantization does not guarantee that the runtime exposes checkpoint-compatible UUIDs. strict_device_check=False removes physical UUID/serial fields from identity comparison only after topology capture; it does not bypass UUID availability, implement device remapping, or promise bit-exact results on different hardware.

Continuation format and failure checks

ContinuationCodec.VERSION is 2; the quantization adapter version is 3, store version is 1, and scheduler continuation version is 1 at this revision. These are different schemas.

Supported values are finite scalar values, dense materialized strided CPU/CUDA tensors, supported torch.device metadata, exact InputCache instances, dictionaries with string/integer/boolean keys, lists, and tuples. The codec rejects cyclic structures, unsupported objects/devices/layouts, and nonfinite scalar values. It does not pickle Python objects or deserialize executable code.

Tensor data is cloned to CPU for safetensors storage while preserving the original device tag in metadata. Restore moves it back to that exact device and verifies shape, dtype, byte equality, and placement. The adapter validates its state schema and reconstructs explicitly supported packed modules, checks tensor dimensions/schemas and dense aliases, restores scheduler state and RNG, and verifies restoration. A checkpoint directory is therefore coupled to a supported execution identity, not a portable interchange format.

Packed reconstruction also has explicit method-specific metadata in checkpoint_modules.py:

Packed module case Additional schema information
General BaseQuantLinear Input/output dimensions, serialized bits, group size, activation ordering, symmetry, bias presence
EXL3 kind="exl3", output dtype, and each saved buffer's shape/dtype
ParoQuant kind="paro" and rotation/runtime-dtype options
QQQ accelerator kernel kind="qqq" and whether group scales exist
BitsAndBytes Compute dtype; 4-bit quantization-state buffer shape, which depends on serialized values rather than dimensions alone
GPTQ/AWQ BitBLAS Compute dtype from the kernel's quantization config

The constructors are explicit code, not import paths supplied by checkpoint data. The storage layer uses filelock; continuation/artifact serialization uses safetensors. Filesystem fsync and atomic replacement are part of publication: the store does not silently promise host-crash durability on a filesystem that cannot satisfy those operations.

10. Weight-only lifecycle

RTNConfig, GGUFConfig, FP8Config, and BitsAndBytesConfig select the weight-only lifecycle. BaseQModel._quantize_weight_only() ignores supplied calibration, rejects LoRA/EoRA, GPTAQ, and FOEM, and creates one WeightOnlyProcessor plus a WeightOnlyLooper. Batch/concatenation/sort arguments do not create activation work here.

WeightOnlyLooper.loop():

  1. Validates the concrete supported config type and optional embedding targets.
  2. Disables use_cache, resolves layers, and materializes direct meta-backed source tensors as needed.
  3. Optionally offloads base modules and builds sequential/flattened module groups.
  4. Creates an optional local ModulePreProcessor for configured source/padding plans.
  5. Starts boundary extensions with a layer/lm-head LoopPlan.
  6. Applies per-layer preparation/conversion; resolves named modules and dynamic exclusions before scheduling.
  7. Calls quantize_module() with a per-module config clone and assigned device, then submodule_finalize() and optional offload.
  8. Publishes completed layer boundaries, restores cache behavior in finally, marks the model quantized, and returns logs.

RTN computes reconstructed weights plus q_scales, q_zeros, and q_g_idx, then uses the selected packer. GGUF/FP8/BitsAndBytes use pack_original(...) directly on the original dense source instead of synthesizing GPTQ Hessians or GPTQ scales; their reconstruction-error log entry is updated after packing. See GGUF, FP8 and NVFP4, and BitsAndBytes.

Input/output embedding fast conversion on this path currently supports RTN only. Boundary extensions with embedding targets are rejected. embed_only disables disk offload and marks the result for replacement-only saving. Tied lm_head weights and unsupported output-head module types are rejected explicitly. Dynamically excluded trailing layers can be skipped without materializing unnecessary quantization work.

11. Native and dequantization passes

NativeProcessor is prepended for GPTAQ or positive-alpha FOEM. For each layer, it merges the module subsets into one capture plan and forwards the cached calibration batches through that plan, rather than repeating collection for each subset. This is not one forward for the entire dataset or model. It places the captured unquantized module inputs in shared module state as native_inp. Its staging device comes from the GPTAQ/FOEM config ("auto" uses DEVICE_1; "cpu" uses CPU). Later processing consumes these pristine inputs before finalization releases them. It does not emit a packed linear.

DequantizeProcessor is not selected by ordinary quantize() method dispatch. BaseQModel._eora_generate() composes it before EoraProcessor, using a mapping of existing quantized modules. It removes each mapped module, optimizes it, dequantizes/transposes its weight to the working device, handles TP padding, and stages wq plus the float32 difference w_wq_diff. It does not by itself export a newly dense model. Its finalizer drops temporary original/reconstructed weight state after consumers finish. See EoRA and Looper Processors.

Errors, edge cases, and operational guidance

Failure or observation Where / cause Action
Missing/invalid quantize config Public entry requires BaseQuantizeConfig Load with a supported quantization config
Already quantized model Ordinary quantize() rejects it unless an embedding request applies Use the explicit supported conversion/requantization workflow; do not call the ordinary path blindly
Missing calibration Calibration lifecycle chosen with calibration=None Supply data or intentionally select a weight-only config
Empty/invalid calibration Processor/helper length, shape, mask, tokenizer, or type validation Check prepared data and ensure length filtering leaves usable examples
Calibration minimum argument appears ineffective Public argument is not forwarded in this revision Account for the documented implementation caveat; do not rely on the warning's suggested call alone
Stop-forward exception during capture The internal sentinel intentionally aborts the unused model tail Do not catch/suppress arbitrary forward errors as if they were this sentinel
Missing/weak expert coverage Routing sent few/no calibration tokens to an expert Inspect coverage, routing policy, fallback, and dynamic exclusions; improve representative calibration
EXL3 missing Hessian/no samples Processor validates capture before solving Fix capture/coverage; ordinary fallback is not a blanket substitute
Device/Hessian assertion Scheduled weight/task/current-device context differs Investigate pool selection and custom model hooks; do not move tensors behind the looper's assignment state
OOM despite disk offload Current Hessians, inputs, replicas, optimization state, and unfinished finalizers remain live Reduce batches/replicas/expert chunks; consider synchronous finalizers and stage-end GC; inspect host memory as well as VRAM
Finalizer failure Packing/offload worker raised Treat the run as incomplete; only a committed quiescent checkpoint is a durable resume point
GPTQMODEL_RESUME=1 Retired interface, rejected explicitly Unset it and use CheckpointConfig in a new checkpoint directory
Checkpoint unsupported feature/chain Entry support validator or adapter allowlist Use the supported subset; do not assume ordinary quantization support implies resume support
Checkpoint identity mismatch Source/config/calibration/runtime/kernel/topology/plan changed Recreate the exact execution identity or use a new root; do not edit manifests to bypass validation
Another writer owns root Nonblocking exclusive file lock Ensure the old run has stopped and released its lease; do not launch concurrent writers
resume="required" with no generation No published checkpoint Select the correct root or start intentionally with "auto"/"never"
resume="never" with published generation Fresh-run policy protects existing work Use a new directory
Corrupt retained checkpoints No complete published candidate survives validation Restore storage from backup or restart in a new root
More GPUs but no speedup Subset barriers, GIL, replication overhead, method restrictions, or kernel occupancy Profile end-to-end; visibility and worker count alone do not establish useful parallelism

docs/exl3-mi350x-quant-lifecycle.md is a historical MI350X/gfx950 measurement record, explicitly not a current implementation specification. Its single-device timing/occupancy results are not universal defaults or throughput promises. In particular, merging sequential subsets can change quantized outputs: preserve numerical semantics when tuning scheduling.

Source files

All paths below are relative to the repository root. The source tree is available at the revision linked in the overview.

Path Main responsibility
gptqmodel/models/base.py quantize, dataset delegation, processor selection, weight-only branch, model hooks, adapter-generation composition
gptqmodel/utils/calibration.py Calibration normalization, validation, filtering, concatenation, sorting, batching
gptqmodel/quantization/config.py Lifecycle selection, execution/memory defaults, fallback, preprocessors
gptqmodel/looper/module_looper.py Calibration orchestration, device pools, named modules, final waits
gptqmodel/looper/stage_inputs_capture.py First-layer capture and lazy module-path materialization
gptqmodel/looper/stage_layer.py Layer/processor/subset flow, replay, finalizers, offload, boundaries
gptqmodel/looper/stage_subset.py Resolved subset plans, coverage, MoE chunks, capture and worker dispatch
gptqmodel/looper/forward_executor.py Serial/parallel cached forward execution
gptqmodel/looper/input_cache.py Batch-aligned replay cache schema
gptqmodel/looper/named_module.py Stable module identity, synchronized state, stream-to-CPU helpers
gptqmodel/looper/execution_state.py Device scheduler continuation
gptqmodel/looper/loop_processor.py Processor policy, state, logging, continuation, hooks
gptqmodel/looper/module_preprocessor.py Ordered source/TP/smoothing plan annotations
gptqmodel/looper/linear_mode.py Fused-linear runtime mode enum
gptqmodel/nn_modules/qlinear/torch_fused.py, torch_fused_awq.py Consumers of LinearMode
gptqmodel/looper/extension.py Execution plans, contexts, borrowed boundaries, extension dispatch
gptqmodel/looper/checkpoint.py, checkpoint_store.py, checkpoint_devices.py, checkpoint_modules.py Boundary persistence, transactional objects, topology, packed schemas
gptqmodel/looper/gptq_checkpoint.py, continuation.py Quantization adapter, identity, session wrapper, safe continuation encoding
gptqmodel/looper/weight_only_looper.py, weight_only_processor.py Calibration-free quantization and packing
gptqmodel/looper/native_processor.py, dequantize_processor.py Pristine-input capture and adapter-generation reconstruction
gptqmodel/utils/offload.py, gptqmodel/utils/model.py Disk offload, graph replacement, packer dispatch
docs/looper-checkpoint-redesign.md Checkpoint design and validation record; implementation restrictions remain authoritative
docs/multi_gpu_quantization.md Logical GPU pools and free-threaded MoE configuration
docs/exl3-mi350x-quant-lifecycle.md Historical quantization-lifecycle profiling, not an API contract

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Using a model

Platforms

Internals

Clone this wiki locally