/ai-risk-management

name: ai-risk-management
description: "Apply the NIST AI Risk Management Framework (AI RMF 1.0) and adjacent guidance to AI / ML systems — model lifecycle governance, fairness and bias evaluation, robustness, transparency, accountability, third-party model risk, monitoring for drift, and AI incident response. Broader than prompt-injection (which is the security slice). Use when the user mentions 'AI risk,' 'AI governance,' 'NIST AI RMF,' 'AI compliance,' 'ML governance,' 'model risk management,' 'AI fairness,' 'AI bias,' 'algorithmic accountability,' 'AI Bill of Rights,' 'EU AI Act,' 'AI transparency,' 'model card,' 'AI red team,' 'AI safety,' 'responsible AI,' 'model drift,' 'concept drift,' 'AI monitoring,' 'AI incident,' or needs to assess or govern an AI / ML system."
allowed-tools: Read, Grep, Glob, Bash, Write, WebSearch

AI Risk Management — Beyond Security, the Whole Model Lifecycle

`prompt-injection` covers the AI security slice — attackers manipulating LLM inputs. This skill covers everything else risk-related about deploying AI / ML systems: governance, fairness, robustness, transparency, monitoring, incident response specific to AI failures, third-party model risk, and compliance with the emerging AI regulatory landscape.

The framing is NIST AI RMF 1.0 (released 2023) — the most widely-adopted voluntary framework — plus the regulatory layer (EU AI Act, US executive orders, sector-specific guidance). Use this skill when you are deploying AI features beyond a chatbot wrapper, when a regulator asks "how do you govern your AI," or when something has gone wrong with an AI system in production.

Cross-references: `prompt-injection` for prompt-injection / LLM-specific security attacks; `threat-modeling` for design-time AI risk modeling; `incident-triage` and `breach-patterns` for AI-related incident response patterns; `csf-mapping` for the broader governance frame that AI RMF sits within.

The NIST AI RMF — four functions

Just like the cybersecurity framework, the AI RMF organizes the work into functions. Same shape, different content.

| Function | What it covers | |---|---| | **Govern (GOV)** | Policy, accountability, roles, risk appetite, AI principles, board oversight, governance structures | | **Map (MAP)** | Context — what is the AI system, what does it do, who is impacted, what could go wrong, what are the legal / ethical constraints | | **Measure (MEAS)** | Evaluate the system — fairness, robustness, accuracy, explainability, privacy, security; quantitative + qualitative metrics | | **Manage (MAN)** | Treat the risks — mitigations, monitoring, incident response, decommissioning, ongoing review |

The framework is voluntary but increasingly cited in contracts, RFPs, executive orders, and emerging regulations. Treat it as the lingua franca of AI risk.

Workflow

Step 1 — Inventory AI systems

Before assessment, build the inventory. Most organizations underestimate how much AI they actually deploy.

| Category | Examples | |---|---| | **First-party trained models** | Recommendation engines, fraud detection, churn prediction, internal ML pipelines | | **First-party LLM use** | Customer support chat, content generation, summarization, code generation, embeddings for search | | **Third-party AI features** | Stripe Radar (fraud), GitHub Copilot (code completion), Salesforce Einstein, Notion AI, Linear AI | | **Embedded AI in products you ship** | Suggested responses, smart defaults, AI sorting / ranking | | **AI in HR / hiring** | Resume screening, candidate matching, performance evaluation — high regulatory exposure | | **AI in customer-facing decisions** | Pricing, eligibility, content moderation, ad targeting — high regulatory exposure |

For each, record: vendor (if any), training data source, deployment context, who it affects, the decision it informs, how decisions are reviewed.

Step 2 — MAP: assess the context per system

For each AI system in the inventory, answer:

• **Purpose** — what is this system's stated goal? Does the actual deployment match?
• **Stakeholders** — who interacts with it, who is affected by its decisions, who is in a position to challenge those decisions?
• **Legal / regulatory context** — is this in scope for a specific law? (EU AI Act high-risk categories, US HUD fair-housing rules, EEOC for employment AI, FTC for unfair / deceptive practices, sector laws)
• **Failure modes** — what does "broken" look like? (Wrong answer, biased answer, hallucinated answer, slow answer, expensive answer, refused-to-answer-something-it-should, answered-something-it-should-not)
• **Reversibility** — when this system makes a wrong call, can the decision be undone? (Mortgage denial: hard to undo. Spam filter: easy)

Step 3 — MEASURE: evaluate the system

The categories of evaluation, with the engineering hooks for each:

Accuracy / performance

• Test set evaluation — held-out data, not the training data
• Performance on slices of data, not just aggregate (the system that's 95% accurate overall may be 60% accurate on the demographic that's most impacted)
• Confusion matrices for classification; quantile-based error analysis for regression
• For LLMs: task-specific evals (HELM, MMLU, custom evals) — and *especially* custom evals on the application's actual prompts

Fairness / bias

• **Demographic parity** — does the system produce similar outcomes across protected classes?
• **Equalized odds** — are false-positive and false-negative rates similar across groups?
• **Calibration** — when the system says "80% likely," is that actually 80% across all groups?
• **Individual fairness** — do similar inputs produce similar outputs?

These metrics often conflict — you cannot maximize all of them simultaneously. The MAP step should have decided which is most important for the use case. For hiring AI, equalized odds matters more than demographic parity. For loan approval, the choice depends on whose interests dominate.

**Tooling:** Fairlear