Skip to content

Loading and Inference

Jump to bottom
Qubitium edited this page Oct 29, 2026 · 4 revisions

Loading and Inference

Documentation home · Quickstart · Lifecycle

On this page

Overview

GPTQModel.load() is the public entry point for saved quantized checkpoints, dense inference/evaluation, and source models prepared for quantization. It inspects the checkpoint and dispatches to from_quantized() or from_pretrained(); these two paths have different device, tokenizer, adapter, and backend behavior. Supplying a QuantizeConfig does not itself quantize weights: quantization is a separate model.quantize(...) operation.

This reference describes source revision c1976918. See Installation and Quickstart for installation and Quantization Lifecycle for source-model preparation and quantization.

Workflow at a glance

Loading intents and dispatch. Checkpoint detection distinguishes dense, source, and saved quantized models.

Reading the diagram: Alternative intents, not sequential phases. Detected quantized inputs normally take from_quantized even when QuantizeConfig is supplied; native-float source exceptions are documented in the page.

Text version of the workflow
  • Dense inference: Dense checkpoint → No QuantizeConfig → HF model placement → Native generation.
  • Prepare quantization: Source checkpoint → Supply QuantizeConfig → from_pretrained → Ready to quantize.
  • Saved quantized model: Recognized metadata → from_quantized → Initialize runtime → Runtime generation.

Open full-size SVG

Concepts

Concept Meaning
Source model A non-quantized model, or a supported native float-format model being used as the input to a new quantization run. Loaded through the model definition's from_pretrained().
Saved quantized model A checkpoint recognized from quantization metadata or a quantization configuration file. Loaded through from_quantized().
Method Quantization algorithm/family recorded in configuration, such as gptq, awq, or vptq.
Format Saved tensor layout, such as gptq, gptq_v2, gptq_p, gemm, or gguf. It is not interchangeable with a backend name.
Backend Native kernel selector or external execution engine; BACKEND.AUTO asks the selector to choose a compatible implementation.
Profile A source-loading/GGUF loading policy, not a universal speed switch for every saved checkpoint.
Wrapper A model-definition subclass of BaseQModel, containing the native model or external runtime, tokenizer, quantization configuration, and generation dispatch.

GPTQModel.load

The signature in gptqmodel/models/auto.py is:

@classmethod
def load(
        cls,
        model_id_or_path: Optional[str],
        quantize_config: Optional[QuantizeConfig | Dict] = None,
        device_map: Optional[Union[str, Dict[str, Union[str, int]]]] = None,
        device: Optional[Union[str, torch.device]] = None,
        backend: Union[str, BACKEND] = BACKEND.AUTO,
        profile: Union[str, int, PROFILE] = PROFILE.AUTO,
        trust_remote_code: bool = False,
        allow_remote_adapter: bool = False,
        adapter_revision: Optional[str] = None,
        adapter_sha256: Optional[str] = None,
        **kwargs,
):
    ...

Parameters

Parameter Default Behavior
model_id_or_path Required; annotated Optional[str] Local model directory, Hub repository ID, or supported GGUF artifact path. Strings are stripped. The annotation does not make None a usable model identifier.
quantize_config None QuantizeConfig or a dictionary converted with QuantizeConfig(**quantize_config). Used for source-model loading/re-quantization, not as an override for the saved quantization configuration on the normal quantized path.
device_map None Forwarded to the selected loading path. A dictionary provides module-to-device placements; string policies enter device normalization/automatic placement logic. Source-model and quantized loading implement different placement policies.
device None Requested device, forwarded to either loading path. Source-model quantization placement also involves its QuantizeConfig.
backend BACKEND.AUTO Normalized through normalize_backend(), then passed to the selected path. Explicit incompatible kernels/formats fail validation; selecting an engine does not convert a checkpoint into an arbitrary supported format.
profile PROFILE.AUTO Normalized through normalize_profile(), passed to source loading. It is not passed to the normal from_quantized() dispatch.
trust_remote_code False Model custom-code permission, resolved through resolve_trust_remote_code(). Native Transformers support can cause a caller's True to be safely reduced to False. This is separate from remote adapter download permission.
allow_remote_adapter False Explicit opt-in for a caller-selected remote adapter on a quantized model. Requires an adapter path, pinned commit, and weights digest; it is not a global permission for checkpoint metadata to fetch adapters.
adapter_revision None Immutable 40-hex-character Hub commit for an explicitly selected remote adapter. Branches and tags are not accepted as pins.
adapter_sha256 None 64-hex-character SHA-256 digest of the remote adapter_model.safetensors.
**kwargs Empty Passed to the selected loader after the public dispatcher's normalization. Examples include dtype, adapter on the quantized path, or engine initialization kwargs for an external backend. Not every kwarg is valid for every path. Use the explicit loader methods for a separate tokenizer_trust_remote_code override; load() already supplies that keyword internally.

The initial AutoConfig probe is not a generic passthrough of all **kwargs: it receives the model identifier, resolved model trust, and internal GGUF load kwargs. Do not assume that a loader kwarg such as revision changes every earlier probing step.

Basic quantized inference

from gptqmodel import GPTQModel

model = GPTQModel.load(
    "ModelCloud/Llama-3.2-1B-Instruct-gptqmodel-4bit-vortex-v1",
    device="cuda:0",
)

output_ids = model.generate(
    "The capital city of France is",
    max_new_tokens=32,
    do_sample=False,
)
print(model.tokenizer.decode(output_ids[0], skip_special_tokens=True))

Native causal-model generation normally returns prompt plus continuation. Slice by the tokenized prompt length when only the continuation is wanted. Do not assume the same return type for every external runtime; see Serving.

Source-model loading is not inference-checkpoint detection bypass

from gptqmodel import GPTQModel, QuantizeConfig

model = GPTQModel.load(
    "/path/to/source-model",
    quantize_config=QuantizeConfig(bits=4, group_size=128),
)
# Quantize separately with a representative calibration dataset.
# model.quantize(calibration_dataset, ...)

If the checkpoint is detected as already quantized, load() takes the quantized path even when quantize_config was supplied, except for the specific native float-format source/re-quantization handling.

Dense inference and evaluation without quantization

For an ordinary dense model, quantize_config=None is supported: the model-definition loader calls the HF loader, sets quantized=False, and returns a wrapper without preparing quantization. It uses the supplied device_map, or "auto" when that map is absent/falsy. This path is useful for a non-quantized evaluation baseline; it is not the same as source loading with a quantization configuration, which prepares CPU-resident source weights and a quantization device.

baseline = GPTQModel.load(
    "/path/to/dense-source-model",
    quantize_config=None,
    device_map={"": "cuda:0"},
)

Use device_map to control this HF dense-loading branch: its HF call does not turn the separate device argument into a single-device map. An already-quantized checkpoint still follows quantized detection; None does not force dequantized loading.

How quantized-checkpoint detection works

GPTQModel.load() performs these operations before dispatch:

  1. When a quantization configuration is supplied, initiates best-effort quantization-related prewarming.
  2. Normalizes a GGUF artifact into a model root and internal filename, and strips a string identifier.
  3. Resolves model-code trust and normalizes configuration, backend, and profile inputs.
  4. Loads an AutoConfig where appropriate. A non-directory absolute local artifact is treated specially rather than blindly passed to AutoConfig.
  5. Allows native float-format checkpoints to be treated as source models for re-quantization when that path is requested.
  6. Checks supported config.quantization_config metadata, then searches for standalone quantization configuration files locally or in the remote repository.
  7. Calls from_quantized() for a recognized checkpoint; otherwise calls from_pretrained().

_is_supported_quantization_config

This helper requires config.quantization_config to be a dictionary. It recognizes case-insensitive strings in quant_format, or in method with fallback to quant_method:

Recognized method string METHOD member
gptq METHOD.GPTQ
gguf METHOD.GGUF
fp8 METHOD.FP8
bitsandbytes METHOD.BITSANDBYTES
awq METHOD.AWQ
paroquant METHOD.PARO
qqq METHOD.QQQ
vptq METHOD.VPTQ
exl3 METHOD.EXL3

The helper first checks _activation_quantization_mode(). Metadata indicating unsupported activation-quantized formats raises ValueError with the detected metadata, rather than being silently treated as a dense model. This guard is not a claim that every format from every producer with a recognized method label is loadable.

Standalone configuration discovery

There are two related but different mechanisms:

Stage Files/metadata examined
Public load() detection Supported quantization_config in AutoConfig, or discovery of quantize_config.json / quant_config.json.
Quantized loader's configuration reader QuantizeConfig.from_pretrained() checks quantize_config.json, then quant_config.json, then config.json; for the last file it reads its quantization_config member.

quantize_config.json is the canonical filename (QUANT_CONFIG_FILENAME). Do not confuse it with quant_config.json, which is supported for compatibility, or invent a quantization_config.json requirement. Missing or malformed metadata can fail in file/JSON loading or configuration validation; a filename's existence alone does not prove that its contents or tensors are compatible.

Explicit loader entry points

GPTQModel.from_pretrained

@classmethod
def from_pretrained(
        cls,
        model_id_or_path: str,
        quantize_config: QuantizeConfig,
        backend: Union[str, BACKEND] = BACKEND.AUTO,
        profile: Union[str, int, PROFILE] = PROFILE.AUTO,
        trust_remote_code: bool = False,
        **model_init_kwargs,
) -> BaseQModel:
    ...

This public method normalizes GGUF arguments and the dtype alias, loads AutoConfig, resolves the model definition through check_and_get_model_definition(), and delegates to its loader. It can redirect to from_quantized() when metadata already identifies supported quantization; it is not a forced-dense switch.

That redirect calls from_quantized(model_id_or_path, trust_remote_code=...) only; it does not forward the caller's backend, adapter, placement, or other initialization kwargs. Prefer load() or an explicit from_quantized() call for saved quantized checkpoints.

The public signature requires the quantize_config argument; pass None explicitly for dense loading without quantization preparation. The model-definition implementation in loader.py additionally exposes dtype="auto", device_map=None, and device=None through its implementation signature. Its configuration/model-specific logic controls actual source-model construction. Dynamic per-module quantization produces a warning about HF Transformers support; vLLM/SGLang support discussed by that warning is still conditional on engine, model, format, and fused-module compatibility.

GPTQModel.from_quantized

@classmethod
def from_quantized(
        cls,
        model_id_or_path: Optional[str],
        device_map: Optional[Union[str, Dict[str, Union[str, int]]]] = None,
        device: Optional[Union[str, int]] = None,
        backend: Union[str, BACKEND] = BACKEND.AUTO,
        adapter: Optional[Adapter | Dict] = None,
        trust_remote_code: bool = False,
        allow_remote_adapter: bool = False,
        adapter_revision: Optional[str] = None,
        adapter_sha256: Optional[str] = None,
        **kwargs,
) -> BaseQModel:
    ...

Notice the public device annotation differs from load() and there is no explicit profile argument. The model-definition implementation also has dtype: [str | torch.dtype] = "auto". These are the source signatures, not an assertion that all annotation combinations are supported by every runtime.

The public method:

  1. Normalizes GGUF paths and torch_dtype/dtype kwargs.
  2. Separates tokenizer custom-code trust from model custom-code trust.
  3. Normalizes an adapter object/dictionary.
  4. Finds the appropriate registered model definition.
  5. Normalizes the backend and calls the model definition's from_quantized().

The implementation in ModelLoader then resolves a local checkpoint directory, loads the saved QuantizeConfig, resolves manifest or explicit adapters, validates backend/format compatibility, loads the tokenizer, and constructs the model/runtime. Native paths instantiate quantized modules, load their tensors, perform format-specific conversion/post-initialization, and put the model in evaluation mode. External runtime paths create a runtime wrapper and attach generation dispatch instead of pretending that the engine is a Transformers module.

The native quantized loader rejects unsafe PyTorch .bin checkpoint loading. Use the supported safetensors checkpoint layout or a supported native GGUF path; renaming a .bin file does not make it safetensors. External engines retain their own checkpoint-reader constraints.

Device placement

Input What to expect
device="cuda:0" An explicit GPU target; on ROCm, PyTorch still uses CUDA device spellings. Kernel selection distinguishes CUDA/ROCm capability.
device_map={"": "cuda:0"} Entire model assigned to a single target through the quantized loader.
Explicit module dictionary Passed into the placement path. Keys must match the actual module hierarchy; embeddings, decoder layers, and output head placement matter.
No explicit map Native quantized loading can construct deterministic layer-wise placement across available supported devices. This is not a promise of a memory-optimal Accelerate map.
String map policy On the native quantized path, accepted strings are "auto", "balanced", "balanced_low_0", and "sequential". They use the loader's deterministic layer-wise map, not separate Accelerate balancing algorithms.
External engine Configure the runtime's placement/parallel options as well. SGLang explicitly translates the requested device family/index into engine kwargs, including base_gpu_id; vLLM receives its forwarded engine kwargs rather than the native layer-placement implementation.

Explicit native backends are checked against the requested device set. A kernel supporting one GPU architecture may not support another GPU in a heterogeneous map. Consult Memory Offload and Multi-GPU, Inference Backends, and Kernel Backends Reference.

For source loading with a quantization configuration, setting QuantizeConfig.device together with either device or device_map raises AttributeError. The quantization device selects computation placement, not a request to eagerly load all source weights onto that accelerator.

Backend and profile selection

The public backend enum is gptqmodel.utils.backend.BACKEND, re-exported by gptqmodel. Representative canonical values used by these loading paths are:

Member Value Role
AUTO auto Compatible native backend selection.
GPTQ_TORCH gptq_torch Torch GPTQ implementation.
GPTQ_TRITON gptq_triton Triton GPTQ implementation.
GPTQ_MARLIN gptq_marlin Marlin GPTQ implementation.
GPTQ_BITBLAS gptq_bitblas BitBLAS GPTQ implementation.
AWQ_TORCH awq_torch Torch AWQ implementation.
AWQ_MARLIN awq_marlin Marlin AWQ implementation.
FP8_TORCH fp8_torch Torch FP8 implementation.
GGUF_TORCH gguf_torch Torch GGUF implementation.
GGUF_TRITON gguf_triton Triton GGUF implementation.
VPTQ_TORCH vptq_torch Torch VPTQ implementation.
EXL3_TORCH exl3_torch Torch EXL3 implementation.
VLLM vllm External vLLM runtime.
SGLANG sglang External SGLang runtime.
MLX mlx Checkpoint conversion into an MLX runtime, not Torch MPS execution.

This is not the complete kernel/alias matrix; use Kernel Backends Reference for that reference. Hardware capability, bit width, group size, symmetry, activation ordering, dtype, and checkpoint layout affect compatibility. AUTO does not mean “Marlin on every machine.”

Model loading can impose narrower rules than a kernel class's declared formats. In particular, serialized Marlin/BitBLAS checkpoints select the matching method-specific backend on the local Torch path, with MLX handled separately. See format-specific loader branches before substituting a backend solely from the class tables.

All PROFILE enum values are:

class PROFILE(str, Enum):
    AUTO = "auto"
    FAST = "fast"
    LOW_MEMORY = "low_memory"

For native GGUF Q1_0 source loading, _resolve_native_gguf_profile() resolves AUTO to FAST; _should_use_dense_native_gguf_path() retains the dense HF import path for Q1_0 with FAST. Do not generalize that exception into a persistent dense-weight cache for quantized inference kernels. See GGUF.

GGUF identifier normalization

normalize_model_id_or_path_for_hf_gguf() in gptqmodel/utils/hf.py is shared by the public entry points. It is distinct from quantization-method detection.

Input situation Result
Explicit existing local .gguf file Resolves to its parent model root plus an internal artifact filename.
Directory/Hub repository with one unambiguous GGUF artifact Can infer the artifact and carry the internal filename to HF configuration/model loading.
Multiple GGUF choices Does not select an arbitrary quantization. Use an explicit local artifact path or a repository layout that identifies one artifact.
Repository also containing dense model weight artifacts Single-GGUF inference is suppressed rather than silently replacing the dense checkpoint.
Public non-None gguf_file=... kwarg Rejected with TypeError; provide the artifact/model identifier through the supported input path instead.

The helper patches Transformers GGUF compatibility as needed. GGUF support, native execution, and HF GGUF import are separate paths; see Packing Formats and Compatibility. The private internal filename kwarg is an implementation detail, not a user API.

Tokenizers and generation

Tokenizer ownership

load_hf_tokenizer() routes construction through Tokenicer.load() and returns the native tokenizer. BaseQModel wraps supplied native tokenizers through Tokenicer as well and stores the wrapper on model.tokenizer; the underlying wrapped model's .tokenizer is the native tokenizer, not the Tokenicer wrapper. Reusable tokenizer/chat-template normalization belongs to Tokenicer, not ad-hoc per-loader patches.

Setting Behavior
Supplied tokenizer BaseQModel construction accepts a native PreTrainedTokenizerBase and wraps it through Tokenicer; quantize(tokenizer=...) also has an explicit override. The standard source/quantized loaders construct their tokenizer from the checkpoint. Do not assume an arbitrary load(tokenizer=...) kwarg replaces that construction.
tokenizer_trust_remote_code Separate tokenizer custom-code override in from_pretrained() / from_quantized(). When omitted, the original model-trust request supplies the tokenizer default before model trust may be reduced for native support. load() supplies the original request explicitly; passing another copy through its kwargs collides.
Pad/EOS metadata Generation uses tokenizer padding metadata where available. Incorrect special-token IDs change masking/decoding and can look like a quantization failure.
Chat template Plain string generate() tokenizes text; it does not automatically turn it into a chat conversation. Apply the model's template explicitly.

BaseQModel.generate

def generate(self, inputs=None, **kwargs):
    ...

Generation is wrapped in torch.inference_mode(). When absent from kwargs, pad_token_id is filled from the tokenizer. Dispatch differs by execution path:

Input/path Behavior
Native text string Tokenizes with return_tensors="pt", padding=True, padding_side="left", normalizes the attention mask, and moves inputs to the model device.
Native list of strings Batched tokenization with the same left-padding policy.
Tensor/token mapping Normalizes supported higher-rank attention masks, then delegates to native generation. Unlike the text branch, it does not automatically move caller-provided tensors/mappings to the model device.
String without a tokenizer Raises rather than guessing a tokenizer.
Runtime with _runtime_generate Delegates to the engine-specific generation adapter. Native Transformers' entire generation API is not guaranteed there.

Native generation kwargs are passed to the underlying model. For external vLLM/SGLang, use supported sampling parameters and max_new_tokens, not Transformers' prompt-inclusive max_length. See Serving for the conversion tables and return types.

Explicit chat-template example

messages = [{"role": "user", "content": "Explain weight-only quantization briefly."}]
inputs = model.tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
)
inputs = inputs.to(model.device)
output_ids = model.generate(inputs=inputs, max_new_tokens=64, do_sample=False)
print(model.tokenizer.decode(output_ids[0], skip_special_tokens=True))

Tokenizer versions may return a tensor or a mapping. The native wrapper/server contain compatibility handling, but custom calling code should not assume .shape exists on every template result. For precise prompt comparisons, inspect the rendered template and token IDs before evaluating quantization quality.

BaseQModel.forward() delegates to the wrapped model. External runtimes are generation integrations, not a guarantee that native forward/logits APIs work on the engine object.

Loading adapters safely

Runtime adapters are normalized by normalize_adapter() in gptqmodel/adapter/adapter.py. The registered serialized adapter name is lora; an existing Adapter instance is also accepted. Serialized A/B fields, if supplied, must be actual tensors, not arbitrary serialized executable objects.

from gptqmodel import GPTQModel

model = GPTQModel.from_quantized(
    "/path/to/quantized-checkpoint",
    adapter={"name": "lora", "rank": None, "path": "/path/to/local-adapter"},
    device="cuda:0",
)

rank=None is populated from the loaded LoRA configuration's r. Every other supplied rank, including 0, must equal that r; 0 is not an auto-rank sentinel in Lora.post_init(). Rank patterns can subsequently override the rank for individual modules. The loader reads adapter_config.json and adapter_model.safetensors, matches per-module LoRA A/B weights, and caches adapter config/tensors by source/pin while consuming their weights. The adapter contribution is out += (x @ A) @ B; device/dtype is aligned to the forward input when needed.

Source permissions

Adapter source Policy
Explicit local directory Accepted without remote permissions. Do not pass remote pin options for a local adapter.
Adapter path in checkpoint configuration Must resolve to an existing relative directory inside the checkpoint. Absolute paths, traversal, and directory symlinks are rejected; file symlinks have a narrow same-checkpoint HF cache-blob exception.
Explicit Hub adapter Disabled by default; requires allow_remote_adapter=True, immutable revision, and safetensors SHA-256.
Explicit HTTPS URL Only supported Hugging Face blob URLs are parsed; the URL revision must match the caller's immutable revision. Arbitrary HTTP/file URLs are not a supported adapter source.
Missing explicit remote path Supplying remote permissions/pins without an explicit remote adapter is an error.

An explicit remote adapter can be supplied through load(..., adapter=...) or directly through from_quantized():

model = GPTQModel.from_quantized(
    "/path/to/quantized-checkpoint",
    adapter={"name": "lora", "rank": None, "path": "owner/adapter-repo"},
    allow_remote_adapter=True,
    adapter_revision=immutable_commit,  # 40 hexadecimal characters
    adapter_sha256=weights_sha256,     # 64 hexadecimal characters
)

The variables must contain verified values for that adapter. Both pins are required; checksum mismatch fails loading. A checkpoint manifest cannot grant itself remote access. Remote-adapter options on a non-quantized load() branch raise ValueError. Model trust_remote_code=True does not replace any of these adapter checks.

See EoRA for generating quantization-error recovery adapters and Inference Backends for kernel-specific adapter support.

Environment variables and inference controls

utils/env.py contains the shared boolean parser, not a registry of all settings:

TRUTHFUL = {"1", "true", "yes", "on", "y"}

def env_flag(name: str, default: str | bool | None = "0") -> bool:
    ...

Values are stripped and lowercased; an unset variable uses the supplied default (None means false). A variable present with an empty or unrecognized value is false. Not every module uses this parser: numeric Torch-kernel switches below use int(...), so use 0/1 rather than yes/no there.

Variable Default/interpretation Scope
GPTQMODEL_USE_MODELSCOPE False via env_flag Selects the ModelScope-enabled Hub path. If requested without the package, raises with pip install modelscope; it is not a silent fallback.
GPTQMODEL_PRINT_MODULE_TREE False Opt-in module-tree printing in the loader.
GPTQMODEL_POOL_PREWARM True via env_flag Controls the best-effort device thread-pool prewarm initiated for source quantization loading.
GPTQMODEL_LINEAR_PREWARM True via env_flag Best-effort one-time background accelerator linear-library initialization.
GPTQMODEL_QUANTIZE_PREWARM True via env_flag Best-effort one-time quantization-payload prewarm.
GPTQMODEL_CPU_WORKERS Explicit integer is clamped to at least 1; invalid value falls back Device-thread-pool CPU workers. Default is capped at 12 for GIL builds, 8 for free-threading, with CPU-count-dependent sizing.
DEBUG False via env_flag Package/importer debugging.
GPTQMODEL_LOG_TIMES False via env_flag Logger timing display.
GPTQMODEL_DISABLE_BITBLAS False via env_flag Disables BitBLAS availability in its backend module.
GPTQMODEL_MARLIN_USE_FP32 True via env_flag Marlin implementation's FP32 accumulation flag.
GPTQMODEL_FP32_ACCUM True via env_flag AWQ/related GEMM accumulation paths that import this flag.
GPTQ_TORCH_STREAMING 0, parsed as integer Torch GPTQ bounded streaming dequantization path when its device/layout conditions allow it.
GPTQ_TORCH_STREAM_TILE 512, parsed as integer Column tile size for that streaming path.
GPTQ_TORCH_TRITON_DEQUANT Auto based on Triton dequantizer availability Explicit 0, false, or False disables it; other values still require an available implementation.
GPTQ_TORCH_ROCM_GEMV / GPTQ_TORCH_ROCM_GEMM 1, parsed as integers Enables the corresponding ROCm-specialized Torch GPTQ paths when supported.
GPTQ_TORCH_CACHE_WEIGHTS / GPTQ_TORCH_LOOKAHEAD 0, parsed as integers Legacy compatibility switches only: enabling them warns; dense weight caching/lookahead remains disabled.

Quantized inference weights must remain quantized between uses. A legacy cache switch does not authorize retaining dense or INT4-to-INT8 expanded weights; dequantization temporaries are local to the current operation. Kernel-specific environment controls are documented in Kernel Backends Reference.

Requirements and compatibility boundaries

Component Declared requirement or boundary at this source revision
Python >=3.10 in pyproject.toml. Optional engines may impose stricter version constraints.
Tokenicer tokenicer>=0.0.14 in requirements.txt. Tokenizer normalization can change independently of quantized math.
Triton extra triton>=3.8.0 for Python <3.15. Installing it does not make a CUDA-only kernel run on every device.
CUDA/ROCm Match the chosen kernel and PyTorch build. ROCm uses cuda devices in PyTorch, but has separate availability/capability checks. There is no single universal CUDA/ROCm minimum enforced by GPTQModel.load() itself.
MLX extra mlx>=0.32.4; mlx_lm>=0.31.3 for Python <3.11, otherwise mlx_lm>=0.32.2. The loader performs its MLX conversion/runtime path and requires those optional packages. See MLX Apple Silicon.
vLLM/SGLang Optional engines with their own dependency/hardware matrices and distinct accepted formats. Use a dedicated engine environment; see Serving.

Pitfalls and troubleshooting

  • Already quantized versus source loading: inspect configuration metadata and filenames before assuming a caller-supplied QuantizeConfig will re-quantize a checkpoint.
  • Placement differences: load() forwards device/device_map to either branch, but source-model quantization placement is not identical to saved-model inference placement.
  • Dtype aliases: normalize_torch_dtype_kwarg() translates deprecated torch_dtype to dtype, warns, and rejects conflicting values. Prefer one dtype argument.
  • Unknown kwargs: passthrough is path-specific. An engine option is not automatically a Transformers option, and generation options belong to generate(), not load().
  • Profile: the normal quantized load() branch does not forward it. Set performance/memory expectations using the actual backend and checkpoint format.
  • Tokenizer and prompt quality: establish a non-quantized baseline with identical rendered prompts and input IDs before calling bad generation a quantization regression.
  • Adapters: local paths, checkpoint-contained paths, and explicit remote adapters have intentionally different permissions.
  • Hardware: a recognized method and a valid safetensors file are necessary but not sufficient for a specific kernel/device combination.
  • Serving: the built-in HTTP shim has different limitations from native generation and external-engine servers; read Serving before exposing it.

Source files

Path Key implementation
gptqmodel/models/auto.py Public load, from_pretrained, from_quantized; metadata detection; model-definition selection; trust/adapter forwarding.
gptqmodel/models/loader.py ModelLoader implementations, device placement, configuration/tensor loading, external runtime routing, GGUF profile handling.
gptqmodel/models/base.py Wrapper/tokenizer initialization, generate, forward, serving entry points.
gptqmodel/quantization/config.py QuantizeConfig, METHOD, FORMAT, canonical/compatibility filenames.
gptqmodel/utils/hf.py GGUF normalization, dtype aliases, trust resolution, Tokenicer construction, generation-config compatibility.
gptqmodel/utils/backend.py BACKEND, PROFILE, normalizers.
gptqmodel/utils/importer.py Device normalization and kernel selection.
gptqmodel/adapter/adapter.py Adapter normalization, LoRA application, adapter cache.
gptqmodel/adapter/remote.py Checkpoint containment, explicit remote permission, immutable revision and digest checks.
gptqmodel/utils/env.py Shared truth-value parser.
gptqmodel/utils/modelscope.py, gptqmodel/utils/hub.py Optional ModelScope selection and Hub routing.
gptqmodel/utils/linalg_warmup.py, gptqmodel/__init__.py Prewarm and worker controls.
gptqmodel/nn_modules/qlinear/torch.py Torch inference switches and disabled legacy dense-cache compatibility shims.
pyproject.toml, requirements.txt Python and optional/core dependency declarations.

See also

Documentation home · Quickstart · Troubleshooting · Repository · PyPI

Source baseline: GPT-QModel 7.7.0 / c197691. Maintained by ModelCloud.ai.

Home

Getting started

Quantization methods

Complementary configs

Using a model

Platforms

Internals

Clone this wiki locally