Repository navigation
Model Definitions and Module Tree
Documentation home · Quickstart · Lifecycle
On this page
A model definition tells GPT-QModel how a Hugging Face model is organized, which modules may be quantized, the order in which their activations must be collected, and which model-specific loading and replay hooks are necessary. Definitions are Python subclasses of BaseQModel in gptqmodel/models/definitions/; MODEL_MAP in gptqmodel/models/auto.py selects one using the checkpoint configuration's model_type.
This page describes the source at revision 44f5c0a23b9b1e86b54888087e393b278b9998a0. Architecture names and Transformers layouts change between releases; inspect the definition shipped with the installed GPT-QModel version, not just a checkpoint's marketing name.
The current declaration is named module_tree. There is no BaseQModel.layers_modules_tree declaration in this revision. Similarly, “base modules” and “layer modules” describe computed plans: use get_base_modules(), build_layer_modules(), simple_layer_modules(), and full_layer_modules(), rather than assigning old base_modules or layer_modules lists.
Reading the diagram: The lanes are separate views of one definition. A full-plan entry is not automatically a quantization target; skipped weights remain dense.
Text version of the workflow
- Quantization target view: Select definition → Resolve effective tree → Build simple plan → Process target subsets.
- Capture and planning view: Use the same tree → Build full plan → Prepare capture.
BaseQModel subclasses torch.nn.Module, owns the underlying Transformers model as self.model, and delegates unresolved attributes to that model through __getattr__. forward() delegates to the wrapped model unless a definition overrides it. The class is decorated at the bottom of base.py by factory functions, not Python mixin inheritance:
BaseQModel = ModelLoader(ModelWriter(BaseQModel))ModelLoader installs from_pretrained() and from_quantized(); ModelWriter installs saving methods. GPTQModel itself is a dispatch facade: GPTQModel() raises an error. Load a model using its class methods.
def __init__(
self,
model: PreTrainedModel,
quantized: bool,
quantize_config: Optional[BaseQuantizeConfig],
tokenizer: Optional[PreTrainedTokenizerBase] = None,
qlinear_kernel: nn.Module = None,
load_quantized_model: bool = False,
trust_remote_code: bool = False,
model_local_path: str = None,
turtle_model: Optional[LazyTurtle] = None,
effective_module_tree: Optional[Any] = None,
):Source: gptqmodel/models/base.py.
Important definition attributes and their base-class defaults:
| Attribute | Base default | Contract |
|---|---|---|
module_tree |
None |
Root path, layer-container delimiter, and layer-relative module mapping; may contain multiple complete variants. |
module_tree_overrides |
None |
Mapping from METHOD to a tree override applied for that quantization method. |
lm_head |
"lm_head" |
Output-head path. Quantizing the head is separately controlled by the quantization configuration. |
pre_lm_head_norm_module |
None |
Final normalization applied to cached inputs before head quantization. Some definitions supply alternative paths. |
rotary_embedding |
None |
Optional architecture-specific rotary embedding path. |
layer_modules_strict |
True |
Require planned modules to exist; heterogeneous/optional-layer definitions may set False. |
awq_scale_optimize_shape_dependent_modules |
None |
Optional module names whose AWQ scale search needs architecture-specific shape handling. |
awq_preserve_explicit_position_embeddings |
False |
Opt-in preservation of explicit position embeddings for AWQ. |
shared_input_verified_model_types |
empty frozenset
|
Model types whose explicit input-sharing declarations are verified; not inherited for verification purposes. |
loader |
AutoModelForCausalLM |
HF auto-loader used to construct/load the architecture. |
require_trust_remote_code |
None |
Architecture requirement checked by loaders when native CausalLM support is unavailable. This is not a blanket permission to execute arbitrary remote code. |
require_pkgs |
None |
Package/version requirements checked by check_versions(). |
require_dtype |
None |
Override automatic/user dtype selection when required by the definition. |
require_fast_init |
True |
Passed as _fast_init during source-model loading. |
require_load_processor |
False |
Load and retain an AutoProcessor on the wrapper. |
require_monkeypatch |
False |
Invoke the definition's monkey_patch() during wrapper initialization. |
dynamic_expert_index |
None |
Config attribute containing the fallback expert count. |
dynamic_expert_indices |
None |
Layer-relative expert-prefix to config-field mapping for multiple expert families. |
supported_dense_vram_strategies, supported_moe_vram_strategies
|
[VramStrategy.EXCLUSIVE, VramStrategy.BALANCED] |
Supported placement strategies. |
support_batch_quantize |
True |
Setting False makes quantization force batch_size=1. |
write_shared_kv_cache |
False |
Let a layer publish KV state needed by later replay layers. |
support_offload_to_disk |
True |
Definitions can disable the lazy/offloaded source path. |
supports_desc_act |
[True, False] |
Allowed activation-order settings; a definition can restrict them. |
info |
{} |
Definition notes; the loader prints an optional "notes" entry. |
modality |
[MODALITY.TEXT] |
Metadata, not the quantization target selector. |
ATTENTION_MASKS_DTYPE |
torch.bool |
Calibration mask dtype; e.g. Omni adapters use integer masks. |
ATTENTION_MASKS_REQUIRED_FOR_INPUT |
False |
Require masks during input capture. |
INPUT_EMBEDDING_EXTRA_ARGS |
None |
When truthy, the base capture hook calls model.generate() with these extra arguments instead of model(..., use_cache=...). |
modules_with_direct_meta_tensors |
[] |
Paths owning direct meta parameters/buffers, materialized outside normal traversal. direct_meta_modules=[] is the compatibility alias. |
HF_CONVERSION_MAP_REVERSED |
None |
Runtime-to-checkpoint renaming/conversion rules for lazy materialization. |
hf_conversion_model_type_alias |
None |
Reuse another model type's inferred HF conversion rules. |
defuser_module_paths |
None |
Explicit nested roots to defuse when experts do not live at the top-level model root. |
moe_expert_module_name_prefixes |
[".expert"] |
Expert-name metadata used by model/looper machinery. |
turtle_materialize_tie_weights |
True |
Whether ordinary turtle materialization re-ties weights. |
out_of_model_tensors |
None |
Extra checkpoint files or tensor prefixes to preserve on save. |
quant_override_files |
{} |
Definition-specific files written after quantized save. |
This is the actual Llama declaration from definitions/llama.py:
module_tree = [
"model",
"layers",
"#",
{
"input_layernorm": ("input_layernorm:!:norm",),
"self_attn": ("q_proj:0:in=x", "k_proj:0:in=x", "v_proj:0:in=x", "o_proj:1"),
"post_attention_layernorm": ("post_attention_layernorm:!:norm",),
"mlp": ("gate_proj:0:in=x", "up_proj:0:in=x", "down_proj:1"),
}
]The prefix before the top-level "#" identifies the repeated decoder layer container, here model.layers. The mapping after it contains paths relative to one decoder layer. Insertion order and subset markers describe forward execution/calibration ordering. Do not alphabetically sort a definition.
| Syntax | Meaning | Example / caution |
|---|---|---|
Top-level "#"
|
Separates root/layer-container path from the layer mapping. |
["model", "layers", "#", {...}]. |
| Nested dictionaries | Extend a relative module path. |
{"mlp": {"experts": {...}}}. |
| Leaf tuple/list | Ordered child module specifications. |
"self_attn": ("q_proj:0", "o_proj:1"). |
:<integer> |
Subset/group identifier; first flag satisfying str.isdigit() is used, default 0. |
q_proj:0, k_proj:0, v_proj:0 share a calculation block; o_proj:1 follows. Numeric groups are scoped/offset by nesting, not global layer IDs. |
:! |
Keep a non-quantized entry in the full plan. | Norms, routers, selected convolutions. It is filtered from the simple quantization plan. |
:? |
Capture-only entry; not a quantized weight target. |
mlp:moe:? lets capture/scaling inspect a parent MoE input. Parent inclusion is opt-in via include_capture_only=True. |
:moe |
Mark a MoE root/family for lifecycle discovery. |
mlp:moe. It does not itself enumerate or quantize experts. |
:norm |
Identify a norm eligible for the separate norm-tuning path. |
input_layernorm:!:norm; linear quantization still excludes it. |
:in=<tag> |
Assert sibling modules consume identical activations. |
gate_proj:0:in=x, up_proj:0:in=x; not implied by numeric grouping. |
runtime|checkpoint |
Ordered aliases; first name is the runtime shell path. | Mixtral's mlp|block_sparse_moe and gate_proj|w1. Later names assist checkpoint lookup, not extra quantization targets. |
Nested "#"
|
Expert-index placeholder. |
"experts": {"#": (...)} becomes experts.{expert_index}.*, then concrete expert paths. |
Empty nested key ""
|
Use the current parent path without adding a segment. | Supports dense and sparse MLP alternatives in the same tree. |
Leaf names that repeat the parent's final component refer to the parent itself; e.g. "input_layernorm": ("input_layernorm:!",) does not produce input_layernorm.input_layernorm.
Flags on a dictionary parent describe that parent entry; they are not recursive exclusions of all descendants. For example, "mlp:moe:?" can capture the MoE input while its explicitly declared expert projections remain quantizable. Mark individual router/norm leaves :! when those weights must stay dense.
module_tree can be a list of complete trees, as in Mllama's current model.language_model.layers and legacy language_model.model.layers layouts. extract_layers_node() returns each distinct layer-container path in first-seen order. build_layer_modules() combines variants and deduplicates identical blocks without changing their order.
_resolve_effective_module_tree() deep-copies the class declaration, applies a method override when present, and attempts autodetection only if no declaration exists and a quantization method is available. apply_module_tree_override() merges on a copy, never the class-owned tree. Initialization freezes the result recursively into effective_module_tree and keeps self.module_tree as its compatibility alias. Mutating the effective mapping raises TypeError.
Class-level planning calls use the static declaration; calls through an instance use its selected tree. __getattribute__ wraps the tree-sensitive class methods in a ContextVar binding, and the quantized loader uses _module_tree_context() while planning before wrapper construction. One instance's method override must not affect another model instance.
@classmethod
def full_layer_modules(cls, model_config=None, is_awq_quantize: bool = False, include_capture_only: bool = False):Source: gptqmodel/models/base.py.
@classmethod
def simple_layer_modules(cls, model_config, quantize_config, is_awq_quantize: bool = False, include_capture_only: bool = False):Source: gptqmodel/models/base.py.
@classmethod
def shared_input_plan(
cls,
model_config=None,
quantize_config=None,
is_awq_quantize: bool = False,
) -> SharedInputPlan:Source: gptqmodel/models/base.py.
| API | Result and purpose |
|---|---|
extract_layers_node() |
List of candidate repeated-layer paths. Definitions may override it for HF layout alternatives. |
build_layer_modules(tree, include_capture_only=False) |
Ordered layer-relative blocks with non-quantized markers preserved; expert templates are not expanded here. |
full_layer_modules(model_config=None, is_awq_quantize=False, include_capture_only=False) |
Build blocks and expand expert indices if config/cardinality are available; retain non-quantized entries. MoE expansion handles capture-only blocks specially, as described below. Used by scaling/capture/planning. |
simple_layer_modules(model_config, quantize_config, is_awq_quantize=False, include_capture_only=False) |
Build/expand, remove :!/:? entries and empty blocks, then remove modules excluded by dynamic matching. This is the quantizable plan. |
get_base_modules(model) |
Walk ancestor roots before the layer container, collecting sibling child modules while excluding every declared continuation path. Returns module paths, not module objects. |
get_modules_with_direct_meta_tensors(model) |
Return configured direct-meta paths resolving to modules. get_direct_meta_modules() delegates here. |
generate_layers_modules_tree_simple(node) |
Produce a simplified display/helper representation; not the authoritative quantization planner. |
get_moe_modules() / get_moe_module_name()
|
Discover :moe paths / layer-level MoE-root names. |
get_norm_modules() |
Collect :norm metadata for the norm-tuning machinery. |
For Llama, simple_layer_modules() yields these relative calculation blocks:
[
["self_attn.q_proj", "self_attn.k_proj", "self_attn.v_proj"],
["self_attn.o_proj"],
["mlp.gate_proj", "mlp.up_proj"],
["mlp.down_proj"],
]The full plan additionally retains input_layernorm:! and post_attention_layernorm:! blocks. get_base_modules() is about non-layer siblings needed for execution/materialization; being a base module does not mean it is quantized. Custom multimodal definitions often extend/replace this method to include vision towers/projectors outside the language-model ancestry.
This recursive helper removes parsed flags from strings and keys, choosing the first alias. For a dictionary value that is a tuple/list of strings, it drops that parent entry if every child has ! or ?. If only some children are flagged, it keeps the block and strips flags from all its children; it does not selectively remove the skipped child. It does not expand experts, resolve dynamic rules, check a live model, or construct execution subsets. Its top-level list recursion tests list, whereas the effective runtime tree is frozen to tuples: use a mutable tree copy when requesting this presentation form. Do not use its output to decide which weights to replace.
build_moe_modules_if_need() expands {expert_index} when a definition supplies dynamic_expert_index and a model config is present. The fallback count comes from text_config, otherwise thinker_config.text_config, otherwise the root config. dynamic_expert_indices can select a count by the longest matching expert prefix; absent/malformed mapped fields fall back to the legacy count.
Non-AWQ expansion is projection-major within a block: all expert gate_proj paths, then all up_proj paths, etc. AWQ expansion preserves non-expert segments in forward order and expands expert segments expert-major where family cardinalities match. During MoE expansion, an all-capture-only block is removed from its original position (leaving an empty block); AWQ may attach its entries to a qualifying expert input block. Therefore, include_capture_only=True does not guarantee that a MoE parent remains in its own block in the full plan. Do not independently reorder shared experts and routed experts: replay depends on the declared order.
gptqmodel/models/shared_input.py derives immutable LeafSpec, SharedInputGroup, and SharedInputPlan objects. GPTQ Hessian collection depends on input activations, so verified modules consuming the same input can avoid duplicate collection.
- The default is one group per quantizable module. A common numeric subset is not proof of shared input.
-
:in=xshares only among leaves with the same immediate parent and tag.self_attn:in=xandmlp:in=xremain independent; each expanded expert has its own parent. - Non-quantized and capture-only leaves do not join groups.
- A leaf repeated across tree variants must have identical subset/input/skip metadata; conflicts raise
ValueError. - The concrete definition must list the checkpoint's
model_typein its ownshared_input_verified_model_types. Inheriting a Llama tree or mapping another alias to Llama does not automatically enable reuse for that alias. - Metadata may span subsets, but runtime deduplication is within a subset: each subset elects its own leader.
dedup_followers/dedup_countreflect only same-subset reuse.
probe_shared_inputs(layer, plan, forward) installs pre-hooks, copies captured tensors to CPU, runs the supplied forward under inference mode, removes hooks, and checks call counts, shape, dtype, and exact tensor equality. Its report distinguishes mismatches, unverified groups, undeclared sharing, and missing_modules. ok is a strict alias of fully_verified, not “no mismatch among the experts that happened to run.” Cover unrouted experts deliberately before declaring a model type verified. See tests/module_tree/test_shared_input_cpu_forward.py for the real-forward verification gate.
@classmethod
def load(
cls,
model_id_or_path: Optional[str],
quantize_config: Optional[QuantizeConfig | Dict] = None,
device_map: Optional[Union[str, Dict[str, Union[str, int]]]] = None,
device: Optional[Union[str, torch.device]] = None,
backend: Union[str, BACKEND] = BACKEND.AUTO,
profile: Union[str, int, PROFILE] = PROFILE.AUTO,
trust_remote_code: bool = False,
allow_remote_adapter: bool = False,
adapter_revision: Optional[str] = None,
adapter_sha256: Optional[str] = None,
**kwargs,
):Source: gptqmodel/models/auto.py.
load() normalizes model IDs/GGUF references, config dictionaries, backend/profile values, and trust settings. It checks supported embedded quantization metadata or quantization config files to choose source-model versus quantized loading. With a new quantization config, recognized native FP8/NVFP4 sources are treated as inputs for new quantization rather than automatically as finished inference checkpoints. Remote-adapter options are rejected on the source-model branch.
| Entry point | Use |
|---|---|
GPTQModel.load(...) |
Automatic source/quantized dispatch. A dictionary quantize_config is converted to QuantizeConfig. |
GPTQModel.from_pretrained(model_id_or_path, quantize_config, backend=BACKEND.AUTO, profile=PROFILE.AUTO, trust_remote_code=False, **model_init_kwargs) |
Select a definition and prepare a source model; already-supported quantized configurations may redirect to from_quantized(). |
GPTQModel.from_quantized(model_id_or_path, device_map=None, device=None, backend=BACKEND.AUTO, adapter=None, trust_remote_code=False, allow_remote_adapter=False, adapter_revision=None, adapter_sha256=None, **kwargs) |
Select a definition and load a quantized checkpoint/runtime. |
check_and_get_model_definition(model_dir, trust_remote_code=False, **config_load_kwargs) |
Load AutoConfig, lowercase model_type, return MODEL_MAP[model_type] or BaseQModel for unknown types. |
The table abbreviates type annotations; the copied load() declaration above is the complete public dispatch signature. Do not confuse its device and device_map arguments with quantization worker pools.
gptqmodel/models/loader.py resolves/downloads the local model path, loads and normalizes HF config, registers Defuser fused-block replacements, chooses dtype/device, loads the tokenizer through the HF/Tokenicer integration, and checks definition requirements. Its source-loader entry point calls torch._dynamo.disable() because quantization is not intended to run through Dynamo/compiled graph capture. A user attn_implementation other than "auto" is propagated to nested configs by the attention override helper.
With a quantization config, the source model is prepared on CPU (or as a meta shell), not eagerly placed on the quantization accelerator. Set QuantizeConfig.device for quantization; passing a separate loader device or device_map while that field is set raises AttributeError.
offload_to_disk=True is the config default; offload_to_disk_path=None is replaced by a managed temporary directory when needed. The lazy path builds a shell with build_shell_model(), converts fused structures through Defuser, and calls LazyTurtle.maybe_create() with the tree, runtime/checkpoint mapping, config, and target shell. LazyTurtle lives in gptqmodel/utils/structure.py, not a lazy_turtle.py module.
The checkpoint source retains an index rather than constructing a second full dense CPU model. It reads requested tensors from local safetensors shards; supported pickle checkpoint extensions (.bin, .pt, .pth, .ckpt) are converted once into temporary safetensors. maybe_create() requires a local directory and returns None if construction fails; the offload loader treats a missing lazy source as an error.
shell_module_materialize() serializes access with the wrapper's RLock, supports an explicit module_path, and delegates ordinary materialization to the turtle alias helpers. The turtle uses runtime/checkpoint aliases and reversed HF conversion rules, handles requested subtrees, honors placement exclusions, and can re-tie weights. Modules owning direct meta tensors have a separate materialization entry point. Native encoded sources also have distinct forward and quantization-source decoding paths; do not copy packed source bytes into dense shell parameters.
Definitions with support_offload_to_disk=False disable this path. A recognized meta-shell build failure can fall back to direct CPU HF loading without a turtle; unrelated runtime errors propagate. MiMo mixed sources explicitly reject that fallback because it cannot preserve their source layout. Without a quantization config, the loader uses the ordinary HF loading branch (with device_map or "auto"), subject to native GGUF/profile redirects.
The quantized path restores the checkpoint quantization config, validates backend/format/device compatibility, creates an HF architecture, defuses where needed, and resolves the effective tree before planning replacements. It finds all candidate modules, excludes base modules and ordinary heads, matches the simple layer plan, and consults should_quantize_module(). Quantized embedding endpoints and explicitly quantized heads have separate handling. The checkpoint is then loaded into the appropriate runtime modules and wrapped with quantized=True.
External runtimes and specialized formats branch within this loader; the tree does not guarantee a particular inference kernel. See Loading and Inference, Inference Backends, and Kernel Backends Reference.
BaseQModel._auto_detect_module_tree() allows METHOD.GPTQ, GGUF, FP8, BITSANDBYTES, EXL3, PARO, and VPTQ. It does not provide a general fallback for every quantization method.
It tries these containers in order:
model.layers
language_model.layers
model.decoder.layers
transformer.h
transformer.blocks
layers
blocks
model.blocks
The first must be a nonempty ModuleList, list, or tuple whose first item is a module. It inspects only layer zero for Linear, Conv1d, and Conv2d, looks for attention/MLP-like immediate parents, and builds leaf-name mappings; if no such parent mapping exists it falls back to a root mapping. Matching parents are collected in a set, so this heuristic does not establish forward execution order. It also cannot establish heterogeneous-layer coverage, safe sharing, router exclusions, custom replay kwargs, or multimodal tower boundaries. Failure to resolve any effective tree raises ValueError. A dedicated tested definition is preferable to relying on AutoCompat.
gptqmodel/models/_const.py supplies shared device-family identifiers, not a per-kernel compatibility matrix:
| Enum | Members and serialized values |
|---|---|
DEVICE |
ALL="all", CPU="cpu", CUDA="cuda", XPU="xpu", NPU="npu", MPS="mps", ROCM="rocm". |
PLATFORM |
ALL="all", LINUX="linux", WIN32="win32", DARWIN="darwin". |
| Constant / helper | Contract |
|---|---|
CPU, META
|
torch.device("cpu"), torch.device("meta"). Meta represents a shell without materialized tensor storage. |
CUDA, CUDA_0, XPU, XPU_0, MPS
|
Devices cuda, cuda:0, xpu, xpu:0, and mps. |
NPU, NPU_0
|
Try npu/npu:0; fall back to CPU if Torch cannot construct the device. This does not make CPU an actual NPU. |
ROCM |
torch.device("cuda:0"); PyTorch HIP shares the CUDA device namespace. |
SUPPORTS_MODULE_TYPES |
[nn.Embedding, nn.Linear, nn.Conv1d, nn.Conv2d, transformers.Conv1D]; individual algorithms/kernels apply further restrictions. |
EXPERT_INDEX_PLACEHOLDER |
"{expert_index}". |
DEFAULT_MAX_SHARD_SIZE |
"4GB". |
DEVICE.type, DEVICE.index
|
ROCm reports type "cuda"; CUDA/ROCm/XPU/NPU have default index 0, other enum members return None. |
DEVICE.to_torch_device() |
Make a concrete device using that type/default index. ALL is a selector, not a concrete Torch device to materialize. |
DEVICE.to_device_map() |
Return {"": self}, substituting DEVICE.CUDA for ROCm. |
normalize_device(type_value) |
Normalize a string, enum, integer, or torch.device to a device family, discarding a string's explicit index. Integer input selects the available family in CUDA/XPU/NPU/MPS/CPU order; it is not a preserved device ordinal. |
get_best_device(backend=BACKEND.AUTO) |
Choose CUDA_0, XPU_0, NPU_0, MPS, then CPU by availability. The function does not choose an inference kernel. |
For backend-specific CUDA architecture, ROCm, operating-system, and dtype limitations, use Kernel Backends Reference. The ordinary architecture/loader path here is Torch-based. The Apple Silicon GPTQModel.quantize_mlx() entry point in auto.py instead lazily delegates to gptqmodel.quantization.mlx_native.quantize_mlx; that implementation loads/saves through mlx_lm.utils and selects native MLX GPTQ or MLX-LM AWQ without constructing this Torch wrapper.
def save(
self,
save_dir: str,
safetensors_metadata: Optional[Dict[str, str]] = None,
max_shard_size: Optional[Union[int, str]] = DEFAULT_MAX_SHARD_SIZE,
meta_quantizer: Optional[str] = None,
eora_path: Optional[str] = None,
split_by: Optional[str] = None,
**kwargs,
):Source: gptqmodel/models/base.py.
save() dispatches on self.quantized. Ordinary quantized models use save_quantized(); a model flagged _model_free_weight_only_replacements_only instead uses save_quantized_embeddings(). Unquantized wrappers call the writer's redirecting save_pretrained() (see the warning below).
save() parameter |
Default | Behavior |
|---|---|---|
save_dir |
Required | Output directory, created by the quantized writer. |
safetensors_metadata |
None |
Dictionary whose keys/values are normalized to strings; writer sets format="pt". |
max_shard_size |
"4GB" |
Integer byte count, integer-valued string, or size string; K/M/G/T/P suffixes use powers of 1024. None disables the size limit. |
meta_quantizer |
None |
Optional additional name:version quantizer fingerprint; invalid formats are warned about. |
eora_path |
None |
Optional adapter output override; otherwise the configured adapter path is used. |
split_by |
None |
Only ordinary safetensors output is enabled in this revision. Empty/"none" strings normalize to None; "layer" and all other modes raise ValueError. |
The default maximum shard size is DEFAULT_MAX_SHARD_SIZE = "4GB". Size-based sharding is separate from layer splitting. Although writer.py contains a layer-grouped streaming implementation, SUPPORTED_SPLIT_BY = {None} disables that mode in this revision. The parser's error message still mentions 'layer'; it does not reflect the enabled set. The specialized free-weight-only writer has a narrower option set than the ordinary quantized writer.
The quantized save flow in writer.py:
- Require
self.quantized, normalizeout_of_model_tensors, and collect extra prefix tensors as sources. - Save
quant_log.csvwhen logs are present, record quantizer/config metadata, and derive the export configuration. Already-loaded quantized models can be reconstructed into a save-compatible representation; EXL3 has dedicated tensor-storage handling. - Attach
config.quantization_config, save model/generation configuration using an empty state dictionary, and savequantize_config.json. Runtime attention implementation fields are temporarily stripped and restored. Dynamic rule order is preserved in the serialized model config. - For shell/turtle models, restore remaining meta layers/parameters from the source while avoiding overwriting quantized modules with their original dense weights. Gather a state dictionary that can read offloaded storage.
- Stream ordinary safetensors shards, write an index when multiple files are produced, and clean stale ordinary weight files. Layer-grouped output is currently disabled by the split-mode parser. Kernels without shard support disable the requested sharding. Metadata values are normalized to strings and
format="pt"is set. - Save EoRA adapters if configured; copy requested passthrough tensor files; copy remote Python files when trusted remote code is in use; save tokenizer and then processor.
-
BaseQModel.save()writes definition-levelquant_override_filesafter the quantized writer returns.
The processor is intentionally saved after the tokenizer, because its multimodal chat template must not be overwritten by a text-only tokenizer template. Extra tensor prefixes are merged into the main weight stream/index; explicitly copied safetensors files are copied without modifying that index. Use prefixes when tensors must participate in the indexed model weights.
save_pretrained() is not a dense passthrough on this wrapper. ModelWriter.save_pretrained() warns and redirects to save_quantized(). Consequently, save() on an unquantized wrapper also reaches the quantized guard and fails; it does not silently save an unquantized HF model. Use the wrapped HF model's own APIs when that is the intended operation.
from types import SimpleNamespace
from gptqmodel import QuantizeConfig
from gptqmodel.models.auto import MODEL_MAP
definition = MODEL_MAP["llama"]
print(definition.extract_layers_node())
print(definition.simple_layer_modules(
SimpleNamespace(model_type="llama"), QuantizeConfig(bits=4, group_size=128)
))- Inspect the actual HF configuration, runtime module names, checkpoint keys, and forward implementation. Identify every repeated layer type, dense/sparse alternative, embedding/output endpoint, and special nonpersistent buffer.
- Add a class under
gptqmodel/models/definitions/, inheriting the nearest compatible definition orBaseQModel. Supply the correct loader and explicitmodule_tree, in forward order. Set non-strict matching only when optional/heterogeneous paths justify it. - Mark routers/norms/skipped projections with
:!; declare capture-only parents only where replay/scaling needs them. For MoE supply expert counts, expert placeholders,:moe, appropriate lifecycle hooks, and nested Defuser roots if necessary. - Add the import/export in
definitions/__init__.pyand import the class inauto.py; add each exact supportedmodel_typetoMODEL_MAPinauto.py.SUPPORTED_MODELSis derived from this map, not a second manually maintained list. Version-dependent definitions use explicit import/registration gates. - Handle HF/checkpoint naming differences through aliases or
HF_CONVERSION_MAP_REVERSED; prefer declarative reversed renamings over ad hoc materialization patches. Account for direct-meta buffers, tied weights, and out-of-model tensor prefixes. - Implement only the hooks needed for capture and replay. The default
should_quantize_layer()andshould_quantize_module()returnTrue; override both when a layer subclass must remain dense during calibration and checkpoint reconstruction. - Validate the tree with tiny real forwards and the existing module-tree tests, then test loading, first-layer capture, quantization, saving, and reloading. Include omitted/dense modules in save/reload checks, not just successfully quantized linears.
- Add
:in=metadata only after actual-forward validation and list every verifiedmodel_typeon the concrete class. Update project model documentation when registering support.
For a new model that is genuinely forward/layout compatible with Llama, a minimal starting point is inheritance rather than copying its parser logic. The following names are placeholders, not existing registry entries:
# gptqmodel/models/definitions/new_architecture.py
from .llama import LlamaQModel
class NewArchitectureQModel(LlamaQModel):
shared_input_verified_model_types = frozenset()# gptqmodel/models/definitions/__init__.py
from .new_architecture import NewArchitectureQModel
# gptqmodel/models/auto.py (after importing the definition)
MODEL_MAP["actual_config_model_type"] = NewArchitectureQModelReplace the inherited tree/loader/hooks when the actual architecture differs. Empty verification keeps sharing disabled until real-forward tests establish its safety; it does not prevent ordinary quantization.
Useful hook points:
| Hook | Default / intended customization |
|---|---|
before_model_load(self, model_local_path, load_quantized_model) |
No-op; invoked before HF source construction. |
after_model_load(self, model, load_quantized_model) |
Return the model; model-specific structural/config adjustments. |
pre_quantize_generate_hook_start() / pre_quantize_generate_hook_end()
|
Materialize capture prerequisites and release/offload them afterward. Base start is a no-op. |
prepare_dataset(...) |
Base text calibration normalization; multimodal adapters may build processor outputs instead. |
move_input_capture_example(...), finalize_input_capture_example(...)
|
Move/normalize examples before initial capture. |
run_input_capture(...) |
Base calls model(...), or model.generate(...) when INPUT_EMBEDDING_EXTRA_ARGS is truthy. Overrides can prepare embeddings and call the first language layer directly; capture still stops at its pre-hook. |
capture_first_layer_positional_inputs(...) |
Base captures hidden_states kwarg or first positional argument. |
capture_first_layer_input_kwargs(...) |
Base leaves captured kwargs unchanged. |
prepare_layer_replay_kwargs(...) |
Base returns additional inputs unchanged; override for layer-specific masks/rotary/state. |
update_layer_replay_kwargs_from_output(...) |
Base returns layer-input kwargs unchanged; override for cross-layer replay state. |
pre_quantize(module) / post_quantize(module)
|
Materialization/device handling around layer processing. |
lm_head_pre_quantize_generate_hook(inputs) |
Apply the configured final norm before head quantization. |
The following table lists the literal MODEL_MAP entries and the conditional registrations in this source revision. Several model types intentionally share a definition. The five conditional entries (qwen3_5, qwen3_5_text, qwen3_5_moe, qwen3_5_moe_text, and zdtaichu5_0) are gated by Version(TRANSFORMERS_VERSION) >= Version("5.2.0"). This is a version check, not a caught optional-import failure: an incompatible import can still raise. A registry entry is not a promise of every backend, calibration modality, or quantization algorithm working with that model.
check_and_get_model_definition() lowercases the configuration's model_type before lookup. The literal legacy keys refinedWeb and refinedWebModel below contain capitals and do not match that lowercased lookup; they are documented as source registry entries, not as evidence that those spellings dispatch successfully through the public loader.
- Never quantize every
Linearmerely because it exists. The tree, dynamic rules, per-layer/module hooks, and embedding/head rules jointly determine replacement. - A layer-relative suffix is not a checkpoint key. Runtime aliases and HF conversion rules must agree with Defuser's shell layout.
- A class declaration is shared across instances. Do not mutate it inside a loader or method-specific path; use the effective-tree mechanism.
- Group IDs describe execution, not equal inputs. Incorrect sharing can silently reuse the wrong Hessian.
- Extra model-type aliases need their own shared-input verification even if the same class/tree is reused.
- Alternative layer paths are complete tree variants or explicit resolver overrides, not additional strings inserted into one prefix.
- Lazy capture can execute vision/audio/projector modules outside the decoder tree. These must be materialized even though their weights remain dense.
- A skip during calibration alone is insufficient: quantized loading and saving must also honor it. Mllama's cross-attention hooks demonstrate the two-sided contract.
- Device constants express available families, not a kernel compatibility guarantee.
_const.py's CUDA helper checks for at least one device with compute capability >= 6.0; individual kernels can require newer architectures. ROCm uses PyTorch'scudadevice namespace.
-
gptqmodel/models/base.py— definition contract, tree parsing/overrides, effective-tree isolation, traversal, capture/replay hooks, materialization, save dispatch. -
gptqmodel/models/auto.py—GPTQModel,MODEL_MAP, conditional registrations, automatic source/quantized dispatch. -
gptqmodel/models/definitions/__init__.pyandgptqmodel/models/definitions/*.py— definition registry imports and architecture declarations. -
gptqmodel/models/loader.py— HF source/quantized loaders, requirement checks, Defuser integration, lazy source setup, runtime replacement. -
gptqmodel/models/writer.py— quantized configuration/weight streaming, sharding, extra tensors, tokenizer/processor packaging. -
gptqmodel/models/shared_input.py— metadata collection, grouping, verification probes. -
gptqmodel/models/moe_lifecycle.py— expert replay/lifecycle interfaces. -
gptqmodel/models/_const.py— device/platform enums, supported module types, expert placeholder, shard default. -
gptqmodel/utils/structure.py—LazyTurtleand checkpoint-backed tensor materialization. -
gptqmodel/quantization/config.py— quantization methods, dynamic exclusions, MoE/configuration types. -
gptqmodel/quantization/mlx_native.py— separate MLX quantization entry point and native GPTQ/MLX-LM AWQ dispatch. -
gptqmodel/looper/stage_inputs_capture.py,gptqmodel/looper/stage_subset.py,gptqmodel/looper/module_looper.py— consumers of the definition and replay plans. -
tests/module_tree/— automatic detection, variants, subset order, MoE metadata, and shared-input tests.