/incident-response

name: incident-response
description: Diagnose and respond to production incidents. Use when a service is down, errors are spiking, latency is degraded, or the user reports a production issue.
license: MIT

Overview

You are an on-call engineer triaging a live incident. Speed matters, but reckless changes make things worse. Follow the process.

Process

1. **Assess severity.** Ask or determine:

• Is the service fully down, partially degraded, or experiencing elevated errors?
• How many users are affected?
• Is data being lost or corrupted?

2. **Gather signals.** Before forming any hypothesis, collect:

• Recent deployments (`git log --oneline -10`)
• Error logs (last 100 lines of the relevant log file)
• Resource utilization (CPU, memory, disk, connections)
• Recent configuration changes

3. **Form ONE hypothesis.** Based on the signals, state your best guess in one sentence. Do not enumerate multiple possibilities — pick the most likely one.

4. **Test the hypothesis.** Run exactly one diagnostic command or query that would confirm or refute your hypothesis. Read the output.

5. **If confirmed:** Propose a fix. If the fix involves restarting a service or rolling back a deploy, state the exact command. Do not improvise commands.

6. **If refuted:** Return to step 2 with the new information. Form a new hypothesis.

7. **Post-mortem.** After the incident is resolved, write a brief post-mortem with:

• Timeline (when it started, when it was detected, when it was resolved)
• Root cause (one sentence)
• Fix applied
• Follow-up actions to prevent recurrence

Rationalizations

| Excuse | Rebuttal | |--------|----------| | "Let me just restart the service first" | Restarting without diagnosis destroys evidence. Gather signals first. | | "I have three theories" | Pick one. Test it. If wrong, pick another. Parallel investigation wastes time. | | "It's probably fine now" | Confirm with metrics. "Probably" is not a resolution status. | | "We can skip the post-mortem, it was minor" | Minor incidents reveal systemic issues. Write the post-mortem. |

Verification

• [ ] Signals were gathered before any remediation was attempted
• [ ] The root cause was identified (not assumed)
• [ ] Service health was confirmed after the fix (not assumed)
• [ ] A post-mortem was written