/llm-integration

name: llm-integration
license: MIT
compatibility: "Claude Code 2.1.277+."
description: LLM integration patterns for function calling, streaming responses, local inference with Ollama, and fine-tuning customization. Use when implementing tool use, SSE streaming, local model deployment, LoRA/QLoRA fine-tuning, or multi-provider LLM APIs.
user-invocable: false
metadata:
  owner-agent: "llm-integrator"
  category: mcp-enhancement
  version: "2.0.0"
  author: "OrchestKit"
  complexity: "medium"
  tags: "llm, function-calling, streaming, ollama, fine-tuning, lora, tool-use, local-inference"
allowed-tools: "Read Glob Grep WebFetch WebSearch"

LLM Integration

Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in `rules/` loaded on-demand.

Quick Reference

| Category | Rules | Impact | When to Use | |----------|-------|--------|-------------| | [Function Calling](#function-calling) | 3 | CRITICAL | Tool definitions, parallel execution, input validation | | [Streaming](#streaming) | 3 | HIGH | SSE endpoints, structured streaming, backpressure handling | | [Local Inference](#local-inference) | 3 | HIGH | Ollama setup, model selection, GPU optimization | | [Fine-Tuning](#fine-tuning) | 3 | HIGH | LoRA/QLoRA training, dataset preparation, evaluation | | [Context Optimization](#context-optimization) | 2 | HIGH | Window management, compression, caching, budget scaling | | [Evaluation](#evaluation) | 2 | HIGH | LLM-as-judge, RAGAS metrics, quality gates, benchmarks | | [Prompt Engineering](#prompt-engineering) | 4 | HIGH | CoT, few-shot, versioning, DSPy optimization, ReAct, cost optimization |

**Total: 20 rules across 7 categories**

Quick Start

# Function calling: strict mode tool definition
tools = [{
    "type": "function",
    "function": {
        "name": "search_documents",
        "description": "Search knowledge base",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"},
                "limit": {"type": "integer", "description": "Max results"}
            },
            "required": ["query", "limit"],
            "additionalProperties": False
        }
    }
}]

# Streaming: SSE endpoint with FastAPI
@app.get("/chat/stream")
async def stream_chat(prompt: str):
    async def generate():
        async for token in async_stream(prompt):
            yield {"event": "token", "data": token}
        yield {"event": "done", "data": ""}
    return EventSourceResponse(generate())

# Local inference: Ollama with LangChain
llm = ChatOllama(
    model="deepseek-r1:70b",
    base_url="http://localhost:11434",
    temperature=0.0,
    num_ctx=32768,
)

# Fine-tuning: QLoRA with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B",
    max_seq_length=2048, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)

Function Calling

Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. On Claude keep the `tools` array stable across turns so the prompt cache holds; add tools mid-session with an inline `tool_addition` (beta `inline-tools-2026-09-15`, Claude API only, available on Sonnet 5.5 but not on Sonnet 5) instead of swapping subsets (https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages#mid-conversation-tool-changes). For large tool sets, use a non-deferred tool search tool with `defer_loading: true` on the rest, never `defer_loading` alone (a request cannot have every tool deferred). Forced `tool_choice` (`any` or a named tool) returns a 400 on Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1; use `auto` with `strict: true` tools. Validate all inputs with Pydantic/Zod, and return errors as tool results. <!-- model-recency-ok: per-model contrast, Sonnet 5 lacks what Sonnet 5.5 has (platform docs 2026-09-28) -->

• `calling-tool-definition.md` -- Strict mode schemas, OpenAI/Anthropic formats, LangChain binding
• `calling-parallel.md` -- Parallel tool execution, asyncio.gather, strict mode constraints
• `calling-validation.md` -- Input validation, error handling, tool execution loops

Streaming

Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.

• `streaming-sse.md` -- FastAPI SSE endpoints, frontend consumers, async iterators
• `streaming-structured.md` -- Streaming with tool calls, partial JSON parsing, chunk accumulation
• `streaming-backpressure.md` -- Backpressure handling, bounded buffers, cancellation

Local Inference

Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.

• `local-ollama-setup.md` -- Installation, model pulling, environment configuration
• `local-model-selection.md` -- Model comparison by task, hardware profiles, quantization
• `local-gpu-optimization.md` -- Apple Silicon tuning, keep-alive, CI integration

Fine-Tuning

Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.

• `tuning-lora.md` -- LoRA/QLoRA configuration, Unsloth training, adapter merging
• `tuning-dataset-prep.md` -- Synthetic data generation, quality validation, deduplication
• `tuning-evaluation.md` -- DPO alignment, evaluation metrics, anti-patterns

Context Optimization

Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.

• `context-window-management.md` -- Five-layer architecture, anchored summarization, compression triggers
• `context-caching.md` -- Just-in-time loading, budget scalin