/ptq

name: ptq
description: >-
  Use when the user asks to "quantize a model", "run PTQ", "post-training
  quantization", "NVFP4 quantization", "FP8 quantization", "INT8
  quantization", "INT4 AWQ", "quantize LLM", "quantize MoE", "quantize VLM",
  or needs to produce a quantized HuggingFace checkpoint from a pretrained
  model using ModelOpt. Do NOT use for multi-candidate recipe
  exploration or optimization (use quant-recipe-search).

ModelOpt Post-Training Quantization

Produce a quantized checkpoint from a pretrained model. **Read `examples/hf_ptq/README.md` first** — it has the support matrix, CLI flags, and accuracy guidance.

Use `quant-recipe-search` for multi-candidate recipe exploration or optimization. Use this skill for each selected recipe's PTQ run.

Step 1 — Environment

Use the common skill's `environment-setup.md` and `workspace-management.md`. After completing them you should know:

• ModelOpt source is available
• Local or remote (+ cluster config if remote)
• SLURM / Docker+GPU / bare GPU
• Launcher available?
• Which workspace to use

Step 2 — Is the model supported?

Check the support table in `examples/hf_ptq/README.md` for verified HF models.

• **Listed** → supported, use `hf_ptq.py` (step 4A/4B)
• **Not listed** → read `references/unsupported-models.md` to determine if `hf_ptq.py` can still work or if a custom script is needed (step 4C)

Step 2.5 — Check for model-specific dependencies

If the model uses `trust_remote_code` (check `config.json` for `auto_map`), inspect its custom Python files for imports not present in the container:

grep -h "^from \|^import " <model_path>/modeling_*.py | sort -u

**Known dependency patterns:**

| Import found | Packages to install | | --- | --- | | `from mamba_ssm` / `from causal_conv1d` | `mamba-ssm causal-conv1d` (Mamba/hybrid models: NemotronH, Jamba) |

If extra deps are needed:

• **Launcher (4B)**: set `EXTRA_PIP_DEPS` in the task's `environment` section — `ptq.sh` installs them automatically
• **Manual (4A)**: `unset PIP_CONSTRAINT && pip install <deps>` before running `hf_ptq.py`

Step 3 — Choose quantization format

**First**, check for a model-specific recipe:

ls modelopt_recipes/models/ 2>/dev/null
ls modelopt_recipes/model_type/<model_type>/ptq/ 2>/dev/null  # per-arch; <model_type> from local config.json (Hub ID: AutoConfig.from_pretrained)

If a model-specific recipe exists, prefer `--recipe <path>` — but **inspect its include/exclude patterns** rather than assuming (e.g. for VLMs, confirm the vision tower is actually excluded).

**If no model-specific recipe**, choose a format based on GPU (details in `examples/hf_ptq/README.md`):

• **Blackwell** (B100/B200/GB200): `nvfp4` variants
• **Hopper** (H100/H200) or older: `fp8` or `int4_awq`

Use `--qformat <name>` (e.g., `--qformat nvfp4`). Format definitions: `modelopt/torch/quantization/config.py`. General PTQ recipes in `modelopt_recipes/general/ptq/` correspond to the same formats — `--qformat` is the simpler way to use them.

Before running PTQ, sanity-check the selected qformat/recipe against the model structure. Inspect the recipe's include/exclude patterns and summarize which layer groups will be quantized and approximately how many modules/layers match (attention projections, MLP projections, experts, etc.). If the match count is 0, or far smaller than expected for the model, stop and fix the recipe or ask the user before launching calibration.

**VLMs:** generic `*mlp*`/`*experts*` recipes also match the vision tower (`model.visual.*`); quantizing the ViT silently breaks image benchmarks. Use the `model_type/<model_type>/ptq/` recipe or add `*visual*`/`*vision_tower*` excludes, then verify in Step 5 — see `references/checkpoint-validation.md`.

If the source checkpoint is already quantized and the requested recipe/config reduces quantization coverage, confirm that intent with the user before running. For example, if an FP8 checkpoint is used as input and the recipe excludes some layers so they would fall back to BF16 instead of staying quantized, call out the affected layer groups and ask whether that FP8-to-BF16 fallback is intended.

> NVFP4 can be calibrated on Hopper but requires Blackwell for inference.

Step 4 — Run PTQ

**Goal: checkpoint on disk** (`.safetensors` + `config.json`).

For **listed models** (4A/4B): run full calibration directly (`--calib_size 512`). For **unlisted models** (4C): run a smoke test first (`--calib_size 4`), wait for success, then full calibration.

Recommended calibration datasets

• **Text-only LLM PTQ:** Prefer the representative `nemotron-post-training-v3` blend. `modelopt/torch/utils/dataset_utils.py` expands it to seven registered Nemotron SFT domains. Configure Hugging Face credentials where required.

  python examples/hf_ptq/hf_ptq.py ... \
      --dataset nemotron-post-training-v3

• **VLM PTQ:** Include image-text calibration with `--calib_with_images`. This path uses `nemotron_vlm_dataset_v2` with the current default subsets `sparsetables`, `plotqa_cot`, and `wiki_en`; `examples/hf_ptq/hf_ptq.py` and `modelopt/torch/utils/vlm_dataset_utils.py` are the source of truth.

  python examples/hf_ptq/hf_ptq.py ... \
      --calib_with_images

`--dataset` selects text-only calibration and cannot substitute for multimodal examples. Use `--dataset cnn_dailymail` only as a fallback when representative data is unavailable, such as without gated-data access or with only a local public cache.

Which path?

In README table? ─→ YES ──→ SLURM (local or remote)? ──→ LAUNCHER (4B)
                  │          Local Docker + GPU? ────────→ LAUNCHER (4B)
                  │          Remote Docker (no SLURM)? ──→ MANUAL (4A)
                  │          Bare GPU (local or remote)? → MANUAL (4A)
                  │
                  └→ NOT LISTED ──→ UNLISTED MODEL (4C)

4A — Direct: supported model, manual execution

pip ins