/chaos-engineering

name: chaos-engineering
description: "Use before running a chaos experiment, gameday or fault injection, or after one to write it up. Designs it with steady state and abort criteria in experiment_designer.py, sizes the blast radius with blast_radius_calculator.py and drafts the postmortem."
context: fork
license: MIT
metadata:
  version: 2.9.0
  author: claude-code-skills
  tags: [chaos-engineering, resilience, fault-injection, gameday, sre, reliability, chaos-toolkit, chaos-mesh, litmus, gremlin, aws-fis]
  compatible_tools: [claude-code, codex-cli, cursor, antigravity, opencode, gemini-cli]

Chaos Engineering

Design experiments that surface real weaknesses in production systems — without becoming outages. Most "chaos engineering" attempts skip steady-state measurement, define no abort criteria, and have no blast-radius bound. This skill enforces the discipline that makes chaos experiments safe and useful.

When to use

• Planning a chaos experiment (what to break, where, when, how to abort)
• Calculating blast radius before running the experiment
• Reviewing an existing experiment plan for safety
• Choosing a chaos tool (Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS)
• Writing a chaos experiment postmortem
• Running a Game Day exercise

When NOT to use

• General incident response (use `incident-response`)
• Threat hunting / red-team (use `red-team`, `threat-detection`)
• Performance load testing (different goal — chaos is about failure modes, not capacity)
• Production debugging (chaos discovers weaknesses preemptively, not after-the-fact)

Core principle: chaos without abort criteria is an outage

The 4 Principles of Chaos Engineering (Netflix, 2016):

1. **Build a hypothesis around steady-state behavior.** Not "what breaks?" but "X holds; will it still hold under fault Y?" 2. **Vary real-world events.** Inject realistic failures: kill nodes, slow networks, lose cache, throttle dependencies. 3. **Run experiments in production.** Staging never has the same failure modes. Start small. 4. **Automate experiments to run continuously.** One-off chaos is a press release; continuous chaos is engineering.

Add a fifth: **Define abort criteria up front.** A chaos experiment with no abort criteria is an outage by another name.

Quick start

SKILL=engineering/chaos-engineering/skills/chaos-engineering

# 1. Design an experiment
python "$SKILL/scripts/experiment_designer.py" --target "checkout-svc" --hypothesis "p99 latency stays <500ms" --attack latency --duration-min 15

# 2. Calculate blast radius
python "$SKILL/scripts/blast_radius_calculator.py" --traffic-share 0.05 --user-pop 1000000 --duration-min 15

# 3. Generate postmortem after the experiment
python "$SKILL/scripts/experiment_postmortem.py" --plan experiment.json --result-log results.txt

The 3 Python tools

All stdlib-only. Run with `--help`.

`experiment_designer.py`

Generates a structured experiment plan from inputs. Enforces the required sections (hypothesis, steady-state metric, blast radius, abort criteria, rollback).

python scripts/experiment_designer.py \
  --target "checkout-svc" \
  --hypothesis "p99 latency stays <500ms when payment-svc is slow" \
  --attack latency \
  --magnitude "+200ms" \
  --duration-min 15 \
  --blast-radius "5% of US traffic" \
  --abort-if "p99 > 1000ms OR error_rate > baseline + 1pp"

Outputs a markdown plan with: hypothesis, steady-state, attack, magnitude, duration, blast radius, abort criteria, rollback procedure, monitoring dashboards, and learning question.

`blast_radius_calculator.py`

Computes the blast radius of a planned experiment. Given traffic share + user population + duration, calculates expected affected users, expected error budget burn, and a risk score.

python scripts/blast_radius_calculator.py \
  --traffic-share 0.05 \
  --user-pop 1000000 \
  --duration-min 15 \
  --baseline-availability 0.999 \
  --expected-impact-availability 0.95

Outputs:

• Expected affected users
• Error budget consumed (in minutes of error budget)
• Risk score: GREEN / YELLOW / RED
• Recommendation: PROCEED / REDUCE / ABORT

GREEN = <1% error budget; YELLOW = 1-10%; RED = >10%.

`experiment_postmortem.py`

Produces a structured postmortem from an experiment plan + results. Catches the common postmortem failure modes: no learning recorded, no follow-up actions, blame-laden language.

python scripts/experiment_postmortem.py --plan experiment.json --result-log results.txt

Outputs markdown with: summary, hypothesis (was it confirmed/refuted?), what we learned, what surprised us, follow-up actions with owners, and link to next experiment.

The 7 attack types (taxonomy)

Different attacks reveal different weaknesses. See `references/attack_taxonomy.md` for full detail.

| Attack | What it tests | Tooling | |---|---|---| | **Latency** | Timeouts, retries, circuit breakers | tc, Chaos Mesh `NetworkChaos` | | **Error** | Error handling, fallback paths | Chaos Mesh `HTTPChaos`, Toxiproxy | | **Resource** (CPU, memory, disk) | Saturation handling, autoscaling | Chaos Mesh `StressChaos`, stress-ng | | **Network partition** | Split-brain, consensus, failover | Chaos Mesh `NetworkChaos` partition | | **Dependency failure** | Graceful degradation, fallback | Service mesh fault injection | | **Time** | Clock skew, NTP issues | libfaketime, Chaos Mesh `TimeChaos` | | **Infrastructure** (kill instance) | Auto-recovery, failover | AWS FIS, Chaos Monkey |

Pick the attack that matches the hypothesis. "What happens if X is slow?" → latency. "What happens if X loses network?" → partition.

Tooling chooser

| Tool | Best for | Pricing | Stack | |---|---|---|---| | **Chaos Toolkit** | Lightweight, language-agnostic, JSON experiments | OSS | Any | | **Chaos Mesh** | Kubernetes-native, rich CRDs, in-cluster | OSS | Kubernetes | | **Litmus** | Kubernetes, Argo-integrated, large library | OSS + Enterprise | Kubernetes | | **Grem