GGUF - Quantization Format for llama.cpp

Kheish Compatibility

This skill is repo-local and stays inactive until explicitly activated.

When the original instructions refer to legacy tool names, use these Kheish mappings:

• terminal => bash
• web_extract => web_fetch, plus web_search when discovery is needed
• search_files => grep_search and glob_search
• browser_* tools require a browser-capable surfaced tool or MCP; if none is available, use the closest available surface and say so explicitly

When the instructions mention local helper files, resolve them from ${KHEISH_SKILL_DIR}.

GGUF - Quantization Format for llama.cpp

The GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.

When to use GGUF

Use GGUF when:

• Deploying on consumer hardware (laptops, desktops)
• Running on Apple Silicon (M1/M2/M3) with Metal acceleration
• Need CPU inference without GPU requirements
• Want flexible quantization (Q2_K to Q8_0)
• Using local AI tools (LM Studio, Ollama, text-generation-webui)

Key advantages:

• Universal hardware: CPU, Apple Silicon, NVIDIA, AMD support
• No Python runtime: Pure C/C++ inference
• Flexible quantization: 2-8 bit with various methods (K-quants)
• Ecosystem support: LM Studio, Ollama, koboldcpp, and more
• imatrix: Importance matrix for better low-bit quality

Use alternatives instead:

• AWQ/GPTQ: Maximum accuracy with calibration on NVIDIA GPUs
• HQQ: Fast calibration-free quantization for HuggingFace
• bitsandbytes: Simple integration with transformers library
• TensorRT-LLM: Production NVIDIA deployment with maximum speed

Quick start