/ai-threat-model

name: ai-threat-model
description: "Builds an evidence-bound AI threat model from partial architecture, trust, identity, data-flow, and control information. Use when scoping an AI assessment, reviewing a design, or prioritizing realistic attack paths under uncertainty."
metadata:
  portability: universal
  discipline: engagement-operations
  domain: ai-systems
  tags: [ai, threat-modeling, architecture, trust-boundaries, risk]

AI threat model

Turn incomplete system knowledge into a decision-ready threat model without inventing architecture. Preserve uncertainty as a first-class result and update the model when evidence changes.

When to Use

• Scoping an AI security review or penetration test.
• Reviewing a design that combines models, retrieval, agents, tools, data, and

conventional application or cloud services.

• Prioritizing assessment paths when architecture evidence is incomplete or

contradictory.

When NOT to Use

• Do not treat the model as permission to validate a threat on a live target.
• Do not use a generic AI risk list in place of the system's actual boundaries.
• Do not assign likelihood or impact precision unsupported by evidence.

Workflow

1. State the decision the threat model must support, the assessment scope, and the evidence cutoff date. 2. Build an assumption register. For every unknown needed to continue, record the assumption, why it matters, the evidence that would resolve it, its owner, and the consequence if it is false. 3. Identify crown jewels and safety properties. Include sensitive data, model behavior, tool authority, workload identity, evaluation integrity, deployment artifacts, human approvals, and availability where relevant. 4. Map components and trust zones. Include users, applications, model gateways, model providers, retrieval pipelines, agent runtimes, tools, memory, registries, CI/CD, monitoring, and administrative planes. 5. Draw data, instruction, identity, and artifact flows separately. A component can trust data but not instructions, or accept an artifact without loading it. 6. For each boundary, ask what can cross, who authorizes it, which identity is used, what validation occurs, what logs survive, and what downstream consumer acts on the result. 7. Construct attack-path hypotheses from evidence-backed transitions. Label every step `observed`, `assumed`, `inferred`, or `blocked`; one unknown step prevents the full path from being called demonstrated. 8. Evaluate threats across the AI-specific surfaces:

• instruction and context control;
• retrieval and knowledge integrity;
• agent and tool authority;
• model and artifact supply chain;
• identity and secret propagation;
• output consumption and automation;
• monitoring, evaluation, and human oversight.

9. Prioritize by plausible consequence, exposed preconditions, boundary count, control evidence, and uncertainty. Do not multiply arbitrary numbers to manufacture precision. 10. Convert priorities into bounded review questions, control gaps, evidence requests, and separately authorized validation objectives. 11. Version the model. Record which new evidence confirmed, weakened, blocked, or split each path.

Deliverable

Return a threat-model package containing the system purpose, scope, evidence cutoff, crown jewels, component and trust-zone map, separated flow maps, assumption register, attack-path hypotheses, control observations, prioritized review questions, and change log.

Rationalizations to reject

• **“This is a standard RAG architecture.”** Shared labels do not prove shared

trust boundaries or controls.

• **“The diagram is authoritative because the client supplied it.”** Supplied

architecture is reported evidence and may be stale.

• **“An assumed step is probably true, so the path is complete.”** A path is no

stronger than its weakest unsupported transition.

• **“Map every framework technique.”** Exhaustive taxonomy mapping can obscure

the system-specific decisions the model exists to support.

• **“A low likelihood makes the control gap unimportant.”** Weak evidence for

likelihood should remain uncertainty, not become a reassuring score.

• **“The model is finished once the diagram is drawn.”** A threat model is a

versioned decision artifact, not a static picture.