/sota-llm-engineering

name: sota-llm-engineering
description: >-
  State-of-the-art LLM application engineering rules (mid-2026 baseline) for BUILDING and AUDITING LLM-powered features. Claude should use this skill whenever it is building, modifying, or reviewing anything that calls a language model — chat features, RAG pipelines, agents and tool use, structured extraction, classification, summarization, embeddings/vector search, evals and regression gates, prompt or context engineering, model selection/routing, fine-tuning decisions, or LLM cost/latency/observability work. Trigger keywords: LLM, AI feature, prompt, system prompt, context window, RAG, retrieval, embeddings, vector DB, rerank, chunking, agent, tool use, MCP, multi-agent, evals, golden set, LLM-as-judge, fine-tuning, model selection, routing, structured output, JSON schema, prompt caching, token budget, counting tokens, instruction file size, tokenizer, hallucination, grounding. Covers build-quality only — for prompt-injection/agent-security use sota-code-security rules/08 and sota-sandboxing rules/05.

SOTA LLM Engineering

Purpose

One skill, two modes. The `rules/` files define the mid-2026 baseline for engineering LLM-powered software that is **measured, grounded, bounded, and observable**. In **BUILD** mode you write LLM features that conform to the rules by default. In **AUDIT** mode you hunt for violations and report them as severity-rated findings. The rules are the single source of truth for both.

Framing: an LLM call is a non-deterministic, expensive, latency-heavy RPC to a dependency that changes underneath you. Everything here follows from that — you don't ship logic you can't measure (evals), you don't trust output you didn't validate (structured output, grounding), you don't run loops you can't stop (budgets), and you don't deploy what you can't trace (observability).

**Scope boundary:** this skill owns build quality. Prompt injection, the lethal trifecta, tool-call authorization, and model-output-as-untrusted-data live in `sota-code-security` rules/08; agent sandboxing/isolation lives in `sota-sandboxing` rules/05; PII/regulatory handling lives in `sota-privacy-compliance`. Reference them; don't re-derive them.

**Freshness rule:** model names, prices, context limits, and spec revisions in these files were verified June 2026 and rot fast. When a decision hinges on a specific model/price/limit, re-verify against provider docs (or the project's `claude-api`-style reference skill) before encoding it. Write code version-agnostically: model IDs and parameters in config, never inline.

BUILD mode — eval-first by default

1. **Write the eval before the feature.** Before any prompt or pipeline work, produce a 20+ case golden set with graded criteria (rules/01). A feature without an eval is a prototype; treat requests to "just add an LLM call" as a request for an eval + a call. 2. **Start at the simplest tier.** Single call → structured output → workflow (code-orchestrated, deterministic) → agent. Escalate only when the eval shows the simpler tier failing (rules/04 has the decision gates). 3. **Defaults, not options.** Schema-constrained output with validation + bounded repair (rules/02); hybrid retrieval with a golden retrieval set when RAG is needed (rules/03); stop conditions + token/cost budgets on every loop (rules/04); retries with jittered backoff, model IDs in config, per-call tracing (rules/05). 4. **Structural over disciplinary.** Prefer designs where the bad variant can't be written: typed prompt templates instead of f-string soup, versioned prompts in repo, a single gateway module for all model calls that enforces tracing/budgets/retries centrally. 5. **When requirements force a deviation** (no eval data yet, latency forbids a judge, unbounded agent demanded), implement the rules file's documented mitigation and leave a `LLMENG:` comment stating the residual risk. 6. **Finish with the audit checklists.** Run the relevant rules files' end-of-file checklists against your own diff before declaring done.

AUDIT mode — hunting quality violations

Process:

1. **Map the LLM surface**: every call site (grep `messages.create`, `chat.completions`, `generateContent`, `invoke`, raw `https://api.`), prompt templates, retrieval pipelines, agent loops, eval suites (or their absence), and the config/env that selects models. 2. **Sweep by rules file, prioritized**: 01 (evals — absence is the #1 systemic finding), 04 (unbounded agents), 05 (cost/reliability), 02 (prompt/output hygiene), 03 (RAG), 06 (data lifecycle). 3. **Check the negatives.** Missing controls are findings: no eval suite, no regression gate in CI, no `max_tokens`/iteration cap, no timeout, no fallback, no trace of prompt/completion/tokens, no schema on extraction, no retrieval metrics, judge never validated against humans. 4. **Trace, don't pattern-match.** Confirm the prompt actually renders what you think (run the template), confirm the parser actually handles refusals /truncation/`stop_reason`, confirm the "eval" actually gates merges rather than decorating a dashboard. 5. **Verify, then report.** State the concrete failure scenario (wrong answer shipped, runaway bill, silent regression on model bump). If unverified, say so and rate conservatively.

Severity conventions

| Severity | Criteria | Examples | |---|---|---| | **Critical** | User-facing wrongness or unbounded damage with no detection/limit: consequential output (money, health, legal, code-exec) shipped with no eval and no human gate; agent loop with no iteration/cost cap; auto-upgraded model with no regression gate; prompt/completion logs leaking secrets or PII wholesale | `while True:` tool loop, no budget; prod extraction parsed with regex from free text feeding payments; `model="latest"` with zero evals | | **High** | Silent quality/cost failure likely in normal operation: no eval suite on a shipped LL