/moai-ref-llm-security

name: moai-ref-llm-security
description: >
  AI/LLM defensive security reference: prompt-injection defense, OWASP LLM Top 10
  defensive mapping, MCP and agentic tool-call hardening, training-data poisoning
  detection, model-output validation and guardrails, MITRE ATLAS defensive
  correlation, and NIST AI RMF governance. Agent-extending skill that amplifies
  backend, security, and AI-application engineering with production-grade
  defensive patterns for LLM-backed systems.
  NOT for: offensive techniques (jailbreak authoring, attack-payload crafting,
  red-team exploitation), model training or fine-tuning methodology, prompt
  optimization for capability, web-app OWASP Top 10 (see moai-ref-owasp-checklist),
  or general API design (see moai-ref-api-patterns).

when_to_use: >
  Use when hardening an LLM-backed application or agent against prompt injection,
  designing guardrails or output validation, scoping MCP/tool permissions,
  detecting training-data or model poisoning, mapping a design against the OWASP
  LLM Top 10 or NIST AI RMF, or correlating defenses to MITRE ATLAS techniques.
  Loads as background knowledge for AI-security review, agentic-system design, and
  LLM-application hardening tasks.

user-invocable: false
metadata:
  version: "1.0.0"
  category: "domain"
  status: "active"
  updated: "2026-06-24"
  tags: "llm, ai-security, prompt-injection, guardrails, mcp, agentic, owasp-llm, mitre-atlas, nist-ai-rmf, reference"

# MoAI Extension: Progressive Disclosure
progressive_disclosure:
  enabled: true
  level1_tokens: 100
  level2_tokens: 3000

LLM / AI Defensive Security Reference

Defensive practitioner reference for hardening LLM-backed applications and agents. Every section is framed as defense, hardening, detection, or verification — it describes the misconfiguration, how to detect it, and how to prevent it, never how to exploit it. Cross-domain web-app vulnerabilities live in `moai-ref-owasp-checklist`; API design lives in `moai-ref-api-patterns`.

Target Use

Apply when reviewing or building an LLM-backed system — a chat product, a retrieval-augmented application, an autonomous agent, or an MCP server. The material assumes an untrusted-input threat model: any text the model reads (user turns, retrieved documents, tool results, file contents) may carry adversarial instructions, and any text the model emits may be acted on downstream.

Trust Boundaries in an LLM System

The core defensive insight: an LLM does not distinguish "data" from "instructions" the way a parser does. Treat every text channel that reaches the model as a boundary where adversarial instructions can enter.

| Channel | Entry risk | Primary defense | |---------|-----------|-----------------| | End-user prompt | Direct prompt injection | Instruction-hierarchy enforcement, input screening | | Retrieved documents (RAG) | Indirect prompt injection | Provenance tagging, content isolation, retrieval allowlist | | Tool / function results | Injected instructions in tool output | Treat tool output as untrusted data, re-validate before re-prompting | | System / developer prompt | Leakage, override | Minimize secrets in prompt, assume prompt is recoverable | | Model output | Improper downstream handling | Schema validation, encoding, never auto-execute raw output | | Training / fine-tuning data | Poisoning | Data lineage, provenance verification, canary detection |

OWASP LLM Top 10 — Defensive Mapping

The OWASP Top 10 for LLM Applications (2025 edition) is the canonical risk index for LLM systems. Each row below states the risk, the defensive check, and the hardening control. Citations are for defensive correlation only.

| ID | Risk | Defensive check | Hardening control | |----|------|-----------------|-------------------| | LLM01 | Prompt Injection | Can external text override system instructions? | Instruction-hierarchy enforcement, input/output screening, content isolation | | LLM02 | Sensitive Information Disclosure | Can the model reveal secrets, PII, or system prompt? | Output filtering, minimize secrets in context, response redaction | | LLM03 | Supply Chain | Are model weights, datasets, and plugins from verified sources? | Provenance verification, model/dataset signing, dependency pinning | | LLM04 | Data and Model Poisoning | Could training/fine-tuning data carry adversarial content? | Data lineage, canary artifacts, anomaly detection on training sets | | LLM05 | Improper Output Handling | Is model output executed/rendered without validation? | Schema validation, context-aware encoding, no auto-execution | | LLM06 | Excessive Agency | Does the agent hold more capability/permission than needed? | Least-privilege tool design, human-in-loop for high-impact actions | | LLM07 | System Prompt Leakage | Does the system prompt hold secrets that leak on extraction? | Never store secrets in the prompt; assume the prompt is recoverable | | LLM08 | Vector and Embedding Weaknesses | Can embedding/RAG stores be poisoned or leak cross-tenant? | Per-tenant isolation, embedding-source validation, access control | | LLM09 | Misinformation | Is unverified model output presented as authoritative? | Grounding, citation requirements, confidence signalling | | LLM10 | Unbounded Consumption | Can a request exhaust tokens, cost, or compute? | Rate limiting, token budgets, request-size caps, timeout enforcement |

Prompt-Injection Defense

Prompt injection is the highest-frequency LLM risk (LLM01). Defense is defense-in-depth — no single control is sufficient, so layer the controls below.

Direct injection (the user is the attacker)

| Layer | Control | Purpose | |-------|---------|---------| | Instruction hierarchy | Mark system instructions as highest-priority; instruct the model that downstream text cannot override them | Reduce override success rate | | Input screening | Scan incoming prompts for known injection markers before they reach the model | Detect obvious override attempts | | Privile