/jetson-inference-mem-tune

name: jetson-inference-mem-tune
description: Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson.
version: 0.0.1
license: "Apache-2.0"
metadata:
  author: "Jetson Team"
  tags: [jetson, inference, memory]
  languages: [python]
  data-classification: public

Jetson Inference Memory Tuning

Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU/variant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.

Purpose

Turn a live `jetson-memory-audit` snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.

When to use

• "Which serving stack should I use on Orin Nano 8 GB to run a 7B model?"
• "vLLM is OOMing — what should `--gpu-memory-utilization` and `--max-model-len` be?"
• "Same model, less memory — can I switch from vLLM to llama.cpp?"
• After `jetson-memory-audit` shows a model server is the top NvMap / PSS consumer.

Prerequisites

• Start with a current `jetson-memory-audit/scripts/audit.sh` JSON snapshot from the target Jetson.
• Know the intended workload: `llm-server`, `vlm-server`, `embedding`, or `rag`.
• If the user gives a desired free-memory target, pass it as `--target-mb`; otherwise let the script use SKU defaults.

Available Scripts

| Script | Purpose | Arguments | |--------|---------|-----------| | `scripts/recommend.py` | Reads an audit JSON and emits runtime plus launch-flag recommendations. | `--audit PATH`, `--runtime`, `--workload`, `--target-mb`, `--human`. |

If your agent runtime supports `run_script`, invoke `run_script("scripts/recommend.py", ["--audit", "/tmp/audit.json", "--runtime", "auto", "--workload", "llm-server"])` and summarize the returned JSON. Otherwise run it with `python3` from the repository root.

Instructions

1. Run `jetson-memory-audit/scripts/audit.sh` to capture the device baseline. 2. Run `scripts/recommend.py --audit /tmp/audit.json --runtime auto --workload llm-server --target-mb 6000` to get a JSON of runtime + flag recommendations. 3. The agent presents the suggested runtime and the exact CLI flags. The user (or an outer agent) launches / restarts the server with those flags. 4. Re-run the audit to verify.

Expected workflow

Use `scripts/recommend.py` for the specific prompt and answer from the JSON it emits. If direct execution is blocked, run it as `python3 {baseDir}/scripts/recommend.py ...`.

• For vLLM OOM prompts, run with `--runtime vllm --workload llm-server` and include concrete `--gpu-memory-utilization=<0.x>` and `--max-model-len=<number>` values from `launch_flags`.
• For "lowest memory" or Orin Nano 8 GB prompts, run with `--runtime auto --workload llm-server`; prefer the runtime in the JSON and explicitly mention the GGUF / 4-bit tradeoff when it selects `llama-cpp`.
• For SGLang prompts, run with `--runtime sglang` and quote `--mem-fraction-static`, `--max-running-requests`, and any context/KV-cache note.
• For "switch from vLLM to llama.cpp" prompts, run with `--runtime llama-cpp` and quote `-ngl`, `-c`, and `--no-mmap`.

Limitations

• Recommendations are only as fresh as the audit JSON. Re-run `jetson-memory-audit` after stopping services, changing power mode, or restarting model servers.
• The script estimates memory pressure from SKU defaults and audit totals; model-specific KV-cache, quantization, and tokenizer behavior can still require benchmarking.
• This skill emits flags only. It does not start, stop, or restart model servers.

Error handling

• Exit `2`: the audit JSON could not be read, parsed, or did not contain valid numeric memory fields. Ask the user to rerun `jetson-memory-audit/scripts/audit.sh`.
• Exit `3`: unsupported runtime or workload request. Re-run with one of the `--runtime` and `--workload` values listed in `scripts/recommend.py --help`.
• Empty or missing `launch_flags`: do not invent fallback flags. Report the script failure and ask for a fresh audit or a supported runtime.

Output contract for `recommend.py`

{
  "sku": "orin-nx",
  "variant": "orin-nx-16gb",
  "mem_total_gb": 16,
  "runtime": "vllm",
  "rationale": "Highest throughput at this memory budget given continuous batching + paged attention.",
  "launch_flags": [
    "--gpu-memory-utilization=0.55",
    "--max-model-len=4096",
    "--max-num-seqs=8",
    "--enable-prefix-caching"
  ],
  "alternatives": [
    { "runtime": "llama-cpp", "rationale": "Lower memory floor with GGUF Q4_K_M.", "launch_flags": ["-ngl 28", "-c 4096", "--no-mmap"] }
  ],
  "notes": ["Lower --gpu-memory-utilization further if you also run a small VLM alongside."]
}

Runtimes covered

| Runtime | Best for | Key memory knobs | Preferred install path | |----------------------|-----------------------------------------------------------|--------------------------------------------------------------------------------------------|------------------------| | **llama.cpp** | Tightest budget; GGUF; Orin Nano-class | `-ngl`, `-c`, `--mlock`, `--no-mmap` | `ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-{orin,thor}` | | **vLLM** | High-throughput serving with continuous batching | `--gpu-memory-utilization`, `--max-model-len`, `--max-num-seqs`, `--enable-prefix-caching` | Thor and Orin JetPack 7.2 / L4T r39+: upstream vLLM 0.20+ (`vllm/vllm-openai`) container or validated native vLLM 0.20+. Older Orin: NVIDIA-AI-IOT image | | **SGLang** | Programmable workflows (RAG,