LiteRT LLM Calibration & Quantization Skill

LiteRT LLM Calibration & Quantization Skill

A skill to guide the calibration, alignment, precision protection, and static range quantization of LiteRT LLMs (such as Gemma 3 1B) for high-performance NPU deployments.

Triggers and Anti-Triggers

• Use when:

  • The user wants to calibrate a LiteRT model against conversational prompt datasets.
  • The user needs to merge calibration JSON files from multiple different tasks.
  • The user wants to statically quantize a model suite (e.g. A8W8 or A16W8 configurations).
  • The user needs to protect RMS Norm or residual paths from quantization noise in Float32.
  • The user needs to align KV Cache quantization scaling parameters across main and auxiliary models.
  • The user wants to run CPU/NPU inference testing on unquantized or quantized LiteRT models.

• Don't use for:

  • Quantizing non-LiteRT models (e.g. PyTorch native, JAX native, or ONNX models).
  • General quantizations that do not involve LLM prefill/decode subgraphs or KV caches.

Principles & Mathematical Gotchas

Every LLM NPU quantization task has highly sensitive numerical paths. You must strictly adhere to these verified gotchas to prevent quality degradation or loop crashes: