Ready to supercharge your AI agents?

Install any skill in seconds. Free to start, no credit card required.

GPTQ (Generative Pre-trained Transformer Quantization) · openlair | DecimalAI

openlair/gptq

SkillScore91Maintained

Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with <2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.

AI/MLby openlair

Links this skill to your agents and keeps it current with the author’s updates.

Automatic

AI Engineer pack

1 · One-time — install the CLI (docs ↗)

pip install decimalai

2 · Get the skill — writes .claude/skills/openlair-gptq/SKILL.md

decimalai skills pull openlair-gptq --out .claude/skills/

No CLI? Get the same file — curl · raw file

curl -fsSL --create-dirs -o .claude/skills/openlair-gptq/SKILL.md https://app.decimal.ai/s/openlair-gptq/SKILL.md

Raw file: /s/openlair-gptq/SKILL.md ↗ · machine-readable summary at /s/openlair-gptq.json ↗

In productionGlobal

how this skill behaves on real traffic · last 30 days

No production data for this skill yet — usage numbers appear here once agents start running it.

Anonymized aggregate across every org using this skill — proof from real production use, not a demo.

Does this skill help? · Benchmark: with vs. withoutsign in to run this on your agents

23 test cases from the publisher’s eval.yaml · each run twice on gemini-3.6-flash — with the skill injected, then without · last run 51d ago

—+186% tokens0% turns91% of whole cases pass with skill (21/23)

Test case	Without → With	Effect	Δ tokens	Δ turns
case-03	✗→✓	▲ Improved	93%	0%
case-04	✗→✓	▲ Improved	219%	0%
case-08	✗→✓	▲ Improved	244%	0%
case-17	✗→✓	▲ Improved	137%	0%
case-18	✗→✓	▲ Improved	207%	0%

Showing 5 of 23 cases

SkillSafety

✓ 3 of 3 passed

Code scan15 of 15 checks clean

0 critical · 0 warning · 0 info — scanner v4 — 8/2/2026

live_secretremote_code_executionreverse_shelldata_exfiltrationssrf_metadataagent_snoopinganti_refusaldestructive_commandsobfuscated_unicodeunicode_homoglyphopaque_blobsuspicious_urltool_overreachtrigger_abuseprompt_injection_phrasing

IntentSafe

LLM judge · does the skill do what it claims

ContentClean

LLM judge · policy & harmful-content review

Source

Imported

OpenLAIR / dr-claw

skills/optimization/gptq/SKILL.md

View on GitHub ↗

License MITImported 8/2/2026Version v1Bundled 3 files

Skill contents

references/

SKILL.md

GPTQ (Generative Pre-trained Transformer Quantization)

Post-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.

When to use GPTQ

Use GPTQ when:

Need to fit large models (70B+) on limited GPU memory
Want 4× memory reduction with <2% accuracy loss
Deploying on consumer GPUs (RTX 4090, 3090)
Need faster inference (3-4× speedup vs FP16)

Use AWQ instead when:

Need slightly better accuracy (<1% loss)
Have newer GPUs (Ampere, Ada)
Want Marlin kernel support (2× faster on some GPUs)

Use bitsandbytes instead when:

Need simple integration with transformers
Want 8-bit quantization (less compression, better quality)
Don't need pre-quantized model files

Quick start

Installation

bash
# Install AutoGPTQ
pip install auto-gptq

# With Triton (Linux only, faster)
pip install auto-gptq[triton]

# With CUDA extensions (faster)
pip install auto-gptq --no-build-isolation

# Full installation
pip install auto-gptq transformers accelerate

Load pre-quantized model

python
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM

# Load quantized model from HuggingFace
model_name = "TheBloke/Llama-2-7B-Chat-GPTQ"

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=False  # Set True on Linux for speed
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Generate
prompt = "Explain quantum computing"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0]))

Quantize your own model

python
from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from datasets import load_dataset

# Load model
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Quantization config
quantize_config = BaseQuantizeConfig(
    bits=4,              # 4-bit quantization
    group_size=128,      # Group size (recommended: 128)
    desc_act=False,      # Activation order (False for CUDA kernel)
    damp_percent=0.01    # Dampening factor
)

# Load model for quantization
model = AutoGPTQForCausalLM.from_pretrained(
    model_name,
    quantize_config=quantize_config
)

# Prepare calibration data
dataset = load_dataset("c4", split="train", streaming=True)
calibration_data = [
    tokenizer(example["text"])["input_ids"][:512]
    for example in dataset.take(128)
]

# Quantize
model.quantize(calibration_data)

# Save quantized model
model.save_quantized("llama-2-7b-gptq")
tokenizer.save_pretrained("llama-2-7b-gptq")

# Push to HuggingFace
model.push_to_hub("username/llama-2-7b-gptq")

Group-wise quantization

How GPTQ works:

Group weights: Divide each weight matrix into groups (typically 128 elements)
Quantize per-group: Each group has its own scale/zero-point
Minimize error: Uses Hessian information to minimize quantization error
Result: 4-bit weights with near-FP16 accuracy

Group size trade-off:

| Group Size | Model Size | Accuracy | Speed | Recommendation | |------------|------------|----------|-------|----------------| | -1 (per-column) | Smallest | Best | Slowest | Research only | | 32 | Smaller | Better | Slower | High accuracy needed | | 128 | Medium | Good | Fast | Recommended default | | 256 | Larger | Lower | Faster | Speed critical | | 1024 | Largest | Lowest | Fastest | Not recommended |

Example:

Weight matrix: [1024, 4096] = 4.2M elements

Group size = 128:
- Groups: 4.2M / 128 = 32,768 groups
- Each group: own 4-bit scale + zero-point
- Result: Better granularity → better accuracy

Quantization configurations

Standard 4-bit (recommended)

python
from auto_gptq import BaseQuantizeConfig

config = BaseQuantizeConfig(
    bits=4,              # 4-bit quantization
    group_size=128,      # Standard group size
    desc_act=False,      # Faster CUDA kernel
    damp_percent=0.01    # Dampening factor
)

Performance:

Memory: 4× reduction (70B model: 140GB → 35GB)
Accuracy: ~1.5% perplexity increase
Speed: 3-4× faster than FP16

High accuracy (3-bit with larger groups)

python
config = BaseQuantizeConfig(
    bits=3,              # 3-bit (more compression)
    group_size=128,      # Keep standard group size
    desc_act=True,       # Better accuracy (slower)
    damp_percent=0.01
)

Trade-off:

Memory: 5× reduction
Accuracy: ~3% perplexity increase
Speed: 5× faster (but less accurate)

Maximum accuracy (4-bit with small groups)

python
config = BaseQuantizeConfig(
    bits=4,
    group_size=32,       # Smaller groups (better accuracy)
    desc_act=True,       # Activation reordering
    damp_percent=0.005   # Lower dampening
)

Trade-off:

Memory: 3.5× reduction (slightly larger)
Accuracy: ~0.8% perplexity increase (best)
Speed: 2-3× faster (kernel overhead)

Kernel backends

ExLlamaV2 (default, fastest)

python
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_exllama=True,      # Use ExLlamaV2
    exllama_config={"version": 2}
)

Performance: 1.5-2× faster than Triton

Marlin (Ampere+ GPUs)

python
# Quantize with Marlin format
config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False  # Required for Marlin
)

model.quantize(calibration_data, use_marlin=True)

# Load with Marlin
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_marlin=True  # 2× faster on A100/H100
)

Requirements:

NVIDIA Ampere or newer (A100, H100, RTX 40xx)
Compute capability ≥ 8.0

Triton (Linux only)

python
model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device="cuda:0",
    use_triton=True  # Linux only
)

Performance: 1.2-1.5× faster than CUDA backend

Integration with transformers

Direct transformers usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load quantized model (transformers auto-detects GPTQ)
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-13B-Chat-GPTQ",
    device_map="auto",
    trust_remote_code=False
)

tokenizer = AutoTokenizer.from_pretrained("TheBloke/Llama-2-13B-Chat-GPTQ")

# Use like any transformers model
inputs = tokenizer("Hello", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)

QLoRA fine-tuning (GPTQ + LoRA)

python
from transformers import AutoModelForCausalLM
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

# Load GPTQ model
model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Llama-2-7B-GPTQ",
    device_map="auto"
)

# Prepare for LoRA training
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Add LoRA adapters
model = get_peft_model(model, lora_config)

# Fine-tune (memory efficient!)
# 70B model trainable on single A100 80GB

Performance benchmarks

Memory reduction

| Model | FP16 | GPTQ 4-bit | Reduction | |-------|------|------------|-----------| | Llama 2-7B | 14 GB | 3.5 GB | 4× | | Llama 2-13B | 26 GB | 6.5 GB | 4× | | Llama 2-70B | 140 GB | 35 GB | 4× | | Llama 3-405B | 810 GB | 203 GB | 4× |

Enables:

70B on single A100 80GB (vs 2× A100 needed for FP16)
405B on 3× A100 80GB (vs 11× A100 needed for FP16)
13B on RTX 4090 24GB (vs OOM with FP16)

Inference speed (Llama 2-7B, A100)

| Precision | Tokens/sec | vs FP16 | |-----------|------------|---------| | FP16 | 25 tok/s | 1× | | GPTQ 4-bit (CUDA) | 85 tok/s | 3.4× | | GPTQ 4-bit (ExLlama) | 105 tok/s | 4.2× | | GPTQ 4-bit (Marlin) | 120 tok/s | 4.8× |

Accuracy (perplexity on WikiText-2)

| Model | FP16 | GPTQ 4-bit (g=128) | Degradation | |-------|------|---------------------|-------------| | Llama 2-7B | 5.47 | 5.55 | +1.5% | | Llama 2-13B | 4.88 | 4.95 | +1.4% | | Llama 2-70B | 3.32 | 3.38 | +1.8% |

Excellent quality preservation - less than 2% degradation!

Common patterns

Multi-GPU deployment

python
# Automatic device mapping
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-GPTQ",
    device_map="auto",  # Automatically split across GPUs
    max_memory={0: "40GB", 1: "40GB"}  # Limit per GPU
)

# Manual device mapping
device_map = {
    "model.embed_tokens": 0,
    "model.layers.0-39": 0,  # First 40 layers on GPU 0
    "model.layers.40-79": 1,  # Last 40 layers on GPU 1
    "model.norm": 1,
    "lm_head": 1
}

model = AutoGPTQForCausalLM.from_quantized(
    model_name,
    device_map=device_map
)

CPU offloading

python
# Offload some layers to CPU (for very large models)
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-405B-GPTQ",
    device_map="auto",
    max_memory={
        0: "80GB",  # GPU 0
        1: "80GB",  # GPU 1
        2: "80GB",  # GPU 2
        "cpu": "200GB"  # Offload overflow to CPU
    }
)

Batch inference

python
# Process multiple prompts efficiently
prompts = [
    "Explain AI",
    "Explain ML",
    "Explain DL"
]

inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    pad_token_id=tokenizer.eos_token_id
)

for i, output in enumerate(outputs):
    print(f"Prompt {i}: {tokenizer.decode(output)}")

Finding pre-quantized models

TheBloke on HuggingFace:

https://huggingface.co/TheBloke
1000+ models in GPTQ format
Multiple group sizes (32, 128)
Both CUDA and Marlin formats

Search:

bash
# Find GPTQ models on HuggingFace
https://huggingface.co/models?library=gptq

Download:

python
from auto_gptq import AutoGPTQForCausalLM

# Automatically downloads from HuggingFace
model = AutoGPTQForCausalLM.from_quantized(
    "TheBloke/Llama-2-70B-Chat-GPTQ",
    device="cuda:0"
)

Supported models

LLaMA family: Llama 2, Llama 3, Code Llama
Mistral: Mistral 7B, Mixtral 8x7B, 8x22B
Qwen: Qwen, Qwen2, QwQ
DeepSeek: V2, V3
Phi: Phi-2, Phi-3
Yi, Falcon, BLOOM, OPT
100+ models on HuggingFace

References

Calibration Guide - Dataset selection, quantization process, quality optimization
Integration Guide - Transformers, PEFT, vLLM, TensorRT-LLM
Troubleshooting - Common issues, performance optimization

Resources

GitHub: https://github.com/AutoGPTQ/AutoGPTQ
Paper: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)
Models: https://huggingface.co/models?library=gptq
Discord: https://discord.gg/autogptq

Illustrative Want to see what openlair/gptq would put in front of your model? Preview the rendered body and check its trigger phrases — no LLM call, no signup.

Latest benchmark · publisher runVerified

8/7/2026, 19:46 UTC · 23 cases

Measured on 20 comparable cases — below the 22 needed to publish a lift, so this result is shown here but not on the registry card or leaderboard.

Lift

+22%

91% vs 70%

Agent turns

1.0 vs 1.0

Tokens

+186%

4,877.15 vs 1,705.25

fail → pass · 5pass → pass · 16fail → fail · 2

Per-case results (23)

Case	Status	Duration (ms)			Turns			Tokens			Tool calls
Case	Status	Without	With	Δ	Without	With	Δ	Without	With	Δ	Without	With	Δ
case-01 I am writing a Python script to quantize Llama 2 7B using a post-training weight quantization technique for standard production deployment. I want the default recommended 4-bit configuration balance between inference speed and accuracy. Provide the configuration object setup specifying bit width, group size, act-order reordering, and dampening factor.	pass→pass	10,083	4,396	-56%	1	1	0%	1,867	4,563	+144%	0	0	—
case-02 I want to quantize a model for ultra-fast inference on NVIDIA A100 GPUs using the Marlin kernel backend. I am preparing the configuration object for 4-bit quantization. Which specific act-order (desc_act) setting must be used so the quantization output is compatible with Marlin?	pass→pass	6,645	3,906	-41%	1	1	0%	1,207	4,499	+273%	0	0	—
case-03 I need to quantize an LLM with 4-bit post-training quantization, but accuracy is extremely critical for my domain and I am willing to sacrifice a small amount of inference speed and memory. How should group_size, desc_act, and damp_percent be configured compared to standard defaults?	fail→pass	17,076	10,998	-36%	1	1	0%	2,998	5,797	+93%	0	0	—
case-04 I want to achieve maximum memory compression (around 5x reduction) on a model using post-training quantization with 128 group size while mitigating quality loss through activation reordering. Write the configuration parameter values.	fail→pass	19,520	4,414	-77%	1	1	0%	1,446	4,617	+219%	0	0	—
case-05 When loading a pre-quantized 4-bit model on an NVIDIA H100 or A100 GPU for highest throughput generation, which kernel flag option should be enabled in the loader function call for optimal speedup?	pass→pass	10,320	4,865	-53%	1	1	0%	1,988	4,715	+137%	0	0	—
case-06 In group-wise weight quantization, a team is deciding between group sizes 32, 128, and 256. They prioritize inference speed and CUDA kernel throughput over minor accuracy differences. Which group size should they select?	pass→pass	10,981	5,463	-50%	1	1	0%	1,966	4,690	+139%	0	0	—
case-07 I am loading a 4-bit quantized model using the Python loader on CUDA. I want to use ExLlamaV2 as the execution kernel. What keyword arguments should be passed to from_quantized?	pass→pass	10,291	5,920	-42%	1	1	0%	1,992	4,922	+147%	0	0	—
case-08 I am deploying Llama 3 on an RTX 4090 GPU and need post-training quantization. My primary constraint is that accuracy loss must be under 1% compared to FP16, and I plan to run on Ada Lovelace architecture with Marlin kernels. Which post-training quantization method is recommended over GPTQ for this specific constraint?	fail→pass	30,124	9,128	-70%	1	1	0%	1,616	5,565	+244%	0	0	—
case-09 I am fine-tuning a model using Hugging Face transformers. I do not have pre-quantized model files, and I want a simple 8-bit quantization setup with minimal setup overhead directly inside standard transformers. Which quantization library/method should I use?	pass→pass	10,170	6,795	-33%	1	1	0%	1,880	5,075	+170%	0	0	—
case-10 I need to deploy a 7B LLM on an Apple M2 Mac and standard x86 CPU servers without dedicated NVIDIA CUDA GPUs. Which quantization format and inference engine should I choose for CPU and Metal acceleration?	pass→pass	14,449	14,365	-1%	1	1	0%	2,542	6,511	+156%	0	0	—
case-11 I am loading a 70B GPTQ quantized model across two 40GB GPUs using AutoGPTQForCausalLM.from_quantized. How should the max_memory parameter be specified to restrict memory usage to 40GB per GPU?	fail→fail	8,747	7,633	-13%	1	1	0%	1,864	5,168	+177%	0	0	—
case-12 I am loading an extremely large 405B GPTQ model on a server with three 80GB GPUs and 256GB system RAM. Write the max_memory dictionary mapping for from_quantized to offload overflow to system RAM.	fail→fail	12,038	8,486	-30%	1	1	0%	2,345	5,324	+127%	0	0	—
case-13 When running .quantize() on a loaded model, what format and structure must the calibration_data list take when extracted from a HuggingFace text dataset?	pass→pass	12,057	10,579	-12%	1	1	0%	2,390	5,895	+147%	0	0	—
case-14 When setting up a PEFT LoraConfig for fine-tuning a 4-bit GPTQ quantized Llama 2 model, which linear projection target modules are typically targeted for parameter-efficient fine-tuning?	pass→pass	8,127	5,611	-31%	1	1	0%	1,654	4,891	+196%	0	0	—
case-15 Before calling get_peft_model on a 4-bit quantized model loaded via transformers for fine-tuning, which helper function from peft must be called to prepare the model for k-bit training?	pass→pass	3,543	2,132	-40%	1	1	0%	742	4,196	+465%	0	0	—
case-16 A developer wants to run GPTQ models with the Marlin kernel backend. What is the minimum NVIDIA GPU compute capability required for Marlin support?	pass→pass	3,379	1,530	-55%	1	1	0%	603	3,952	+555%	0	0	—
case-17 When evaluating perplexity on datasets like WikiText-2, approximately what percentage of accuracy degradation (perplexity increase) is expected when converting Llama 2 models from FP16 to GPTQ 4-bit with group size 128?	fail→pass	8,414	3,254	-61%	1	1	0%	1,873	4,445	+137%	0	0	—
case-18 I have a single RTX 3090 GPU with 24GB VRAM and need to run inference on a 70B model with roughly 4x memory reduction and minimal accuracy loss. Which post-training 4-bit quantization framework designed for GPU execution should I use?	fail→pass	32,106	11,425	-64%	1	1	0%	1,980	6,083	+207%	0	0	—
case-19 How can a developer load a pre-quantized GPTQ model directly using the Hugging Face transformers library without explicitly calling AutoGPTQForCausalLM?	pass→pass	9,252	6,685	-28%	1	1	0%	1,920	5,122	+167%	0	0	—
case-20 What class from auto_gptq is used to define quantization settings such as bit-width, group size, and activation ordering prior to instantiating AutoGPTQForCausalLM.from_pretrained for quantization?	pass→pass	3,440	2,535	-26%	1	1	0%	708	4,221	+496%	0	0	—
case-21 After running model.quantize(calibration_data) on an AutoGPTQForCausalLM instance, what method calls are used to save the quantized weights locally and publish them to Hugging Face Hub?	pass→pass	7,161	4,307	-40%	1	1	0%	1,368	4,630	+238%	0	0	—
case-22 I am deploying an AutoGPTQ model on Windows Server and want to set use_triton=True. Will this kernel backend work on Windows?	pass→pass	8,682	5,734	-34%	1	1	0%	1,531	4,726	+209%	0	0	—
case-23 Compared to standard 16-bit floating point (FP16), what memory reduction factor does 4-bit GPTQ weight quantization achieve?	pass→pass	3,726	2,463	-34%	1	1	0%	667	4,201	+530%	0	0	—

These cases were also screened by the skill’s intent / safety pass.See the Trust review ↗Reproduce this benchmark in Colab ↗

How this skill was measured

DecimalAI ran this skill against gemini-3.6-flash twice over the same eval suite — once with the skill loaded and once without — and compared the two runs case by case. 23 cases were attempted, and 20 counted toward the lift figure. The other 3 produced results that are not comparable between the two arms, so they are excluded from the headline rather than averaged into it. The headline lift of +22 percentage points is the difference between those two pass rates over the 20 comparable cases.

Model measured: gemini-3.6-flash
Graded by: gemini-3.6-flash
Grading method: judged
Verification: verified
Pass rate without the skill: 69.6%
Pass rate with the skill: 91.3%
Cases attempted: 23
Cases counted toward lift: 20
Cases that regressed: 0
Skill version measured: v1
Run completed: 8/7/2026, 19:46 UTC

Measured example — same prompt, with vs. without the skill · from the benchmark run · case case-03

Prompt

I need to quantize an LLM with 4-bit post-training quantization, but accuracy is extremely critical for my domain and I am willing to sacrifice a small amount of inference speed and memory. How should group_size, desc_act, and damp_percent be configured compared to standard defaults?

Without the skill loaded, the model failed this case. With it loaded, the same prompt on the same model passed. This is one improved case from the latest verified run; every case, including any that regressed, is in the table above.

Show both answers

SkillScore blends 4 signals1 of 4 complete

BenchmarkPass rate WITH skill on a verified run91

Live evalNeeds ≥10 evaluated traces in 30 days

AI ratingNeeds ≥10 rated traces

AdoptionNeeds ≥10 serves across ≥2 orgs in 30 days

Efficiency · separate axis (at equal correctness)+0% turns+207% tokens

Shown for transparency; not credited unless a verified run shows no correctness regression.

Backed by 1 of 4 signals — each added signal makes the score more robust.

Verified on gemini-3.6-flash · re-tested 51 days ago

Other measured skills in the registry, with their headline benchmark lift.