/vllm

name: vllm
description: Deploy and serve LLMs with vLLM — OpenAI-compatible inference server with PagedAttention, continuous batching, and quantization support.
version: 1.0.0
author: hermes-CCC (ported from Hermes Agent by NousResearch)
license: MIT
metadata:
  hermes:
    tags: [MLOps, vLLM, Inference, Serving, OpenAI-Compatible, LLM]
    related_skills: []

vLLM

Purpose

• Use this skill to deploy local or remote LLM inference with `vllm`.
• Prefer it when you need OpenAI-compatible serving, high throughput, and modern GPU utilization.
• vLLM is strongest for decoder-only chat and completion models.
• It is a good default for production inference when latency and token throughput matter.

Install

• Install from PyPI:

pip install vllm

• Verify the install:

python -c "import vllm; print(vllm.__version__)"

• Match your CUDA, NVIDIA driver, and PyTorch stack before deploying on GPUs.
• If deployment is containerized, prefer pinning a known-good image or package version.

What vLLM Gives You

• OpenAI-compatible HTTP API for chat and completion workflows
• PagedAttention for efficient KV cache memory usage
• Continuous batching for higher aggregate throughput
• Tensor parallelism for multi-GPU serving
• Quantization support for lower memory footprints
• Streaming token responses
• Good support for major open-weight model families

Common Models

• `meta-llama/Llama-3.1-8B-Instruct`
• `meta-llama/Llama-3.1-70B-Instruct`
• `Qwen/Qwen2.5-7B-Instruct`
• `Qwen/Qwen2.5-14B-Instruct`
• `mistralai/Mistral-7B-Instruct-v0.3`
• `microsoft/Phi-3-medium-4k-instruct`
• `google/gemma-2-9b-it`

Basic Serve

• Minimal OpenAI-compatible server:

python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct

• By default this serves on port `8000`.
• The API base path is `http://localhost:8000/v1`.
• Health check endpoint:

curl http://localhost:8000/health

Key Flags

• `--port`: bind a non-default server port
• `--tensor-parallel-size`: split one model across multiple GPUs
• `--gpu-memory-utilization`: cap how much GPU RAM vLLM should attempt to use
• `--max-model-len`: set maximum sequence length for inference
• `--quantization`: load quantized weights when supported

Serve Examples

• Serve on port `8080`:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8080

• Raise the context window and tune memory usage:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92

• Use four GPUs with tensor parallelism:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

Quantization

• vLLM commonly works with these quantization paths:
• `awq`
• `gptq`
• `bitsandbytes`
• `fp8`

• Example with AWQ:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct-AWQ \
  --quantization awq

• Example with GPTQ:

python -m vllm.entrypoints.openai.api_server \
  --model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
  --quantization gptq

• Example with FP8:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization fp8

• AWQ and GPTQ reduce VRAM pressure at some accuracy or compatibility cost.
• `bitsandbytes` is useful when using 4-bit or 8-bit HF-compatible flows.
• FP8 is hardware-sensitive and best validated on your actual deployment target.

Call Through the OpenAI Client

• vLLM exposes an OpenAI-style API, so the standard OpenAI Python client is a common fit.
• Point the client at the local base URL:

from openai import OpenAI

client = OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarize why continuous batching matters."},
    ],
    temperature=0.2,
    max_tokens=256,
)

print(response.choices[0].message.content)

• `base_url='http://localhost:8000/v1'` is the key integration setting.
• Many SDK-based applications can switch from OpenAI-hosted inference to vLLM with only the base URL and model name changed.

Curl Chat Example

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are concise."},
      {"role": "user", "content": "Explain PagedAttention in two sentences."}
    ],
    "temperature": 0.2,
    "max_tokens": 128
  }'

Async Engine Usage in Python

• Use the async engine when embedding vLLM inside a Python application instead of only exposing the HTTP server.
• This pattern is useful for custom services, pipelines, or batched internal inference.

import asyncio

from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams


async def main() -> None:
    engine_args = AsyncEngineArgs(
        model="meta-llama/Llama-3.1-8B-Instruct",
        gpu_memory_utilization=0.9,
        max_model_len=8192,
    )
    engine = AsyncLLMEngine.from_engine_args(engine_args)
    sampling_params = SamplingParams(temperature=0.2, max_tokens=128)

    request_id = "req-1"
    prompt = "List three use cases for OpenAI-compatible local inference."

    async for output in engine.generate(prompt, sampling_params, request_id):
        if output.finished:
            print(output.outputs[0].text)


asyncio.run(main())

• Use unique request IDs for concurrent work.
• Reuse one engine per process rather than creating a new engine for every request.
• Keep sampling params explicit for reproducibility in evaluation workflows.

Multi-GPU No