/model-serving

name: model-serving
description: LLM and ML model deployment for inference. Use when serving models in production, building AI APIs, or optimizing inference. Covers vLLM (LLM serving), TensorRT-LLM (GPU optimization), Ollama (local), BentoML (ML deployment), Triton (multi-model), LangChain (orchestration), LlamaIndex (RAG), and streaming patterns.

Model Serving

Purpose

Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.

When to Use

• Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
• Building AI APIs with streaming responses
• Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
• Implementing RAG pipelines with vector databases
• Optimizing inference throughput and latency
• Integrating LLM serving with frontend chat interfaces

Model Serving Selection

LLM Serving Engines

**vLLM (Recommended Primary)**

• PagedAttention memory management (20-30x throughput improvement)
• Continuous batching for dynamic request handling
• OpenAI-compatible API endpoints
• Use for: Most self-hosted LLM deployments

**TensorRT-LLM**

• Maximum GPU efficiency (2-8x faster than vLLM)
• Requires model conversion and optimization
• Use for: Production workloads needing absolute maximum throughput

**Ollama**

• Local development without GPUs
• Simple CLI interface
• Use for: Prototyping, laptop development, educational purposes

**Decision Framework:**

Self-hosted LLM deployment needed?
├─ Yes, need maximum throughput → vLLM
├─ Yes, need absolute max GPU efficiency → TensorRT-LLM
├─ Yes, local development only → Ollama
└─ No, use managed API (OpenAI, Anthropic) → No serving layer needed

ML Model Serving (Non-LLM)

**BentoML (Recommended)**

• Python-native, easy deployment
• Adaptive batching for throughput
• Multi-framework support (scikit-learn, PyTorch, XGBoost)
• Use for: Most traditional ML model deployments

**Triton Inference Server**

• Multi-model serving on same GPU
• Model ensembles (chain multiple models)
• Use for: NVIDIA GPU optimization, serving 10+ models

LLM Orchestration

**LangChain**

• General-purpose workflows, agents, RAG
• 100+ integrations (LLMs, vector DBs, tools)
• Use for: Most RAG and agent applications

**LlamaIndex**

• RAG-focused with advanced retrieval strategies
• 100+ data connectors (PDF, Notion, web)
• Use for: RAG is primary use case

Quick Start Examples

vLLM Server Setup

# Install
pip install vllm

# Serve a model (OpenAI-compatible API)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --dtype auto \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --port 8000

**Key Parameters:**

• `--dtype`: Model precision (auto, float16, bfloat16)
• `--max-model-len`: Context window size
• `--gpu-memory-utilization`: GPU memory fraction (0.8-0.95)
• `--tensor-parallel-size`: Number of GPUs for model parallelism

Streaming Responses (SSE Pattern)

**Backend (FastAPI):**

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI
import json

app = FastAPI()
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

@app.post("/chat/stream")
async def chat_stream(message: str):
    async def generate():
        stream = client.chat.completions.create(
            model="meta-llama/Llama-3.1-8B-Instruct",
            messages=[{"role": "user", "content": message}],
            stream=True,
            max_tokens=512
        )

        for chunk in stream:
            if chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                yield f"data: {json.dumps({'token': token})}\n\n"

        yield f"data: {json.dumps({'done': True})}\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache"}
    )

**Frontend (React):**

// Integration with ai-chat skill
const sendMessage = async (message: string) => {
  const response = await fetch('/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message })
  })

  const reader = response.body!.getReader()
  const decoder = new TextDecoder()

  while (true) {
    const { done, value } = await reader.read()
    if (done) break

    const chunk = decoder.decode(value)
    const lines = chunk.split('\n\n')

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = JSON.parse(line.slice(6))
        if (data.token) {
          setResponse(prev => prev + data.token)
        }
      }
    }
  }
}

BentoML Service

import bentoml
from bentoml.io import JSON
import numpy as np

@bentoml.service(
    resources={"cpu": "2", "memory": "4Gi"},
    traffic={"timeout": 10}
)
class IrisClassifier:
    model_ref = bentoml.models.get("iris_classifier:latest")

    def __init__(self):
        self.model = bentoml.sklearn.load_model(self.model_ref)

    @bentoml.api(batchable=True, max_batch_size=32)
    def classify(self, features: list[dict]) -> list[str]:
        X = np.array([[f['sepal_length'], f['sepal_width'],
                       f['petal_length'], f['petal_width']] for f in features])
        predictions = self.model.predict(X)
        return ['setosa', 'versicolor', 'virginica'][predictions]

LangChain RAG Pipeline

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Qdrant
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load and chunk documents
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50)
chunks = text_splitter.split_documents(documents)

# Create vector store
embeddings = OpenAIEmbeddi