/llama-cpp

name: llama-cpp
description: Run quantized LLMs locally with llama.cpp — CPU+GPU inference, GGUF format, OpenAI-compatible server, and Python bindings.
version: 1.0.0
author: hermes-CCC (ported from Hermes Agent by NousResearch)
license: MIT
metadata:
  hermes:
    tags: [MLOps, llama-cpp, Local-Inference, GGUF, Quantization, CPU]
    related_skills: []

llama.cpp

Purpose

• Use this skill to run quantized GGUF models on laptops, workstations, and edge systems.
• Prefer it when you need portable local inference without a heavyweight serving stack.
• `llama.cpp` is especially useful for CPU-first deployments, low-cost GPU offload, and offline workflows.
• Python users typically access it through `llama-cpp-python`.

Install

• Fastest path for Python users:

pip install llama-cpp-python

• Build from source when you need custom acceleration backends or tighter platform control.
• Source builds are common for CUDA, Metal, ROCm, Vulkan, and CPU-tuned environments.
• Confirm the package imports successfully:

python -c "from llama_cpp import Llama; print('ok')"

Model Format

• `llama.cpp` primarily uses the `GGUF` model format.
• GGUF packages tokenizer metadata, architecture settings, and quantized weights in one artifact.
• Choose a GGUF variant that matches your hardware budget and quality target.

Download GGUF Models

• Hugging Face is the standard source for GGUF checkpoints.
• Common repos include:
• `bartowski/*`
• `TheBloke/*`

• Typical examples:
• `bartowski/Llama-3.1-8B-Instruct-GGUF`
• `TheBloke/Mistral-7B-Instruct-v0.2-GGUF`
• `bartowski/Qwen2.5-7B-Instruct-GGUF`

• Store the downloaded file locally, for example:
• `models/llama-3.1-8b-instruct-q4_k_m.gguf`

Quantization Levels

• `Q4_K_M`: best balance for many local deployments
• `Q5_K_M`: more quality, more RAM or VRAM
• `Q8_0`: highest quality among common quantized options
• `Q2_K`: very small, but quality drops sharply

• Start with `Q4_K_M` unless you already know the task is quality-sensitive.
• Move to `Q5_K_M` or `Q8_0` for coding, reasoning, or long-form generation where quality matters more.
• Use `Q2_K` only for extreme memory constraints or experiments.

Basic Python Usage

from llama_cpp import Llama

llm = Llama(
    model_path="models/llama-3.1-8b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=0,
)

output = llm(
    "Explain why GGUF is useful for local inference.",
    max_tokens=256,
    temperature=0.7,
)

print(output["choices"][0]["text"])

Important Init Parameters

• `model_path`: path to the `.gguf` file on disk
• `n_gpu_layers`: number of transformer layers to offload to GPU
• `n_ctx`: context size, constrained by the model and available memory
• `n_threads`: CPU worker threads for prompt processing and generation

• These four parameters are the first tuning knobs to adjust for almost every deployment.

Initialization Guidance

• Keep `model_path` on a local SSD when possible.
• Set `n_threads` close to the number of performant CPU cores, not necessarily total logical threads.
• Increase `n_ctx` only after checking memory pressure.
• Increase `n_gpu_layers` gradually if the model fails to load or performance is unstable.

Generation Call

• Basic call shape:

result = llm(
    "Write a short checklist for running a local LLM service.",
    max_tokens=256,
    temperature=0.7,
)

• Access the text with:

print(result["choices"][0]["text"])

• Keep `max_tokens` bounded for interactive usage.
• Use lower temperatures for summarization, extraction, and tool-style tasks.

Chat Format

• For instruction-tuned models, prefer the chat API:

from llama_cpp import Llama

llm = Llama(
    model_path="models/qwen2.5-7b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_gpu_layers=20,
    n_threads=8,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise and technical."},
        {"role": "user", "content": "List three tradeoffs of 4-bit quantization."},
    ],
    max_tokens=256,
    temperature=0.4,
)

print(response["choices"][0]["message"]["content"])

• Use the chat API when the model card says the checkpoint is chat-tuned or instruct-tuned.
• Keep prompt templates aligned with the model family if outputs seem malformed.

Streaming Output

• Stream tokens for responsive CLI or web applications:

stream = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise."},
        {"role": "user", "content": "Describe GPU offload in llama.cpp."},
    ],
    max_tokens=128,
    temperature=0.3,
    stream=True,
)

for chunk in stream:
    delta = chunk["choices"][0].get("delta", {})
    text = delta.get("content")
    if text:
        print(text, end="", flush=True)

• Streaming is useful for chat UIs, terminals, and server-sent event bridges.

GPU Offload

• `n_gpu_layers=-1` means full GPU offload when supported by the backend and hardware.
• You can also offload only the first `N` layers:

llm = Llama(
    model_path="models/mistral-7b-instruct-q5_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=-1,
)

• If full offload fails, try a partial value like `20`, `30`, or `40`.
• Partial offload is common on consumer GPUs with limited VRAM.

CPU-Only Example

from llama_cpp import Llama

llm = Llama(
    model_path="models/phi-3-mini-q4_k_m.gguf",
    n_ctx=4096,
    n_threads=12,
    n_gpu_layers=0,
)

• CPU-only mode is viable for smaller models and latency-tolerant tasks.
• It is a good fit for offline assistants, batch summarization, and test environments.

Context Size

• `n_ctx` controls the context window in tokens.
• Larger values increase RAM or VRAM usage.
• The effective maximum depends on the model architecture, quantization, and rope scaling setup.
• Do not assume every GGUF file supports the same long cont