/incident-response

name: incident-response
description: Production incident coordination with emergency triage, RCA, and postmortem generation
argument-hint: [--severity p0,p1,p2] [--skip-triage] [--postmortem]
allowed-tools: Task, Read, Write, Edit, Bash, Glob, Grep, SlashCommand, AskUserQuestion
model: inherit
enabled: true

Incident Response - Production Incident Coordination

You are an expert incident response orchestrator managing production incidents using Tresor's operations and analysis agents. Your goal is to quickly triage, investigate, resolve, and learn from production incidents.

Command Purpose

Coordinate production incident response with:

• **Emergency triage** - Immediate assessment and mitigation
• **Parallel investigation** - Multiple specialists investigate simultaneously
• **Root cause analysis** - Comprehensive RCA with timeline
• **Resolution tracking** - Document steps taken and resolution
• **Postmortem generation** - Blameless postmortem with preventive measures
• **Communication** - Status updates for stakeholders

---

Execution Flow

Phase 0: Incident Classification

**Step 1: Parse Arguments**

const args = parseArguments($ARGUMENTS);
// --severity: p0, p1, p2 (default: ask user)
// --skip-triage: Skip triage phase (if already triaged)
// --postmortem: Generate postmortem after resolution

**Step 2: Incident Assessment**

Ask user to describe the incident:

await AskUserQuestion({
  questions: [{
    question: "What is the incident severity?",
    header: "Severity",
    multiSelect: false,
    options: [
      {
        label: "P0 - Critical",
        description: "Service down, users unable to use product, data loss"
      },
      {
        label: "P1 - High",
        description: "Major functionality broken, significant user impact"
      },
      {
        label: "P2 - Medium",
        description: "Minor functionality broken, limited user impact"
      }
    ]
  },
  {
    question: "What symptoms are you observing?",
    header: "Symptoms",
    multiSelect: true,
    options: [
      { label: "High error rate", description: "500 errors, exceptions in logs" },
      { label: "Service unavailable", description: "Cannot reach service" },
      { label: "Slow performance", description: "Timeouts, high latency" },
      { label: "Data corruption", description: "Incorrect data, missing records" }
    ]
  }]
});

**Step 3: Select Incident Response Team**

Based on severity and symptoms:

function selectIncidentTeam(severity, symptoms) {
  const team = {
    // Phase 1: Emergency Triage (always immediate)
    phase1: {
      required: [
        '@incident-coordinator',  // Lead incident response
      ],
      max: 1,
      duration: '5-10 minutes',
    },

    // Phase 2: Parallel Investigation (3 specialists)
    phase2: {
      required: [
        '@root-cause-analyzer',  // Deep investigation
      ],

      conditional: [
        symptoms.includes('high-error-rate') ? '@backend-reliability-engineer' : null,
        symptoms.includes('slow-performance') ? '@performance-tuner' : null,
        symptoms.includes('database-issues') ? '@database-admin' : null,
        symptoms.includes('infrastructure-issues') ? '@devops-engineer' : null,
        symptoms.includes('security-breach') ? '@security-incident-responder' : null,
      ].filter(Boolean),

      max: 3,  // Up to 3 specialists investigate in parallel
      duration: '20-30 minutes',
    },

    // Phase 3: RCA & Timeline (sequential)
    phase3: {
      required: [
        '@root-cause-analyzer',  // Comprehensive RCA
      ],
      max: 1,
      duration: '30-45 minutes',
    },

    // Phase 4: Postmortem (optional)
    phase4: {
      required: args.postmortem || severity === 'p0' ? [
        '@postmortem-writer',
      ] : [],
      max: 1,
      duration: '20-30 minutes',
    },
  };

  return selectOptimalAgents(team);
}

---

Phase 1: Emergency Triage (Immediate)

**Agent:**

• `@incident-coordinator`

**Purpose:** Immediate assessment and mitigation

**Execution**:

const phase1Results = await Task({
  subagent_type: 'incident-coordinator',
  description: 'Emergency incident triage',
  prompt: `
# Incident Response - Phase 1: Emergency Triage

## Incident Details
- Severity: ${severity}
- Symptoms: ${symptoms.join(', ')}
- Reported: ${timestamp}
- Incident ID: incident-${timestamp}

## Your Task (URGENT - Complete in 5-10 minutes)

### 1. Immediate Assessment

**Gather Critical Information:**
\`\`\`bash
# Check service status
curl -I https://api.example.com/health

# Check error logs (last 15 minutes)
tail -1000 /var/log/app.log | grep ERROR

# Check application metrics
# - Error rate
# - Request rate
# - Response time
\`\`\`

**Quick Assessment:**
- What is failing?
- How many users affected?
- Started when? (approximate time)
- Still ongoing?

### 2. Impact Assessment

**User Impact:**
- Percentage of users affected (all, subset, specific feature)
- Geography affected (all regions, specific region)
- User segments affected (free vs paid, mobile vs web)

**Business Impact:**
- Revenue impact (if payments/transactions affected)
- Data loss risk
- Compliance implications
- Reputational impact

### 3. Immediate Mitigation Options

**Quick Mitigations to Consider:**

**Option 1: Rollback**
\`\`\`bash
# If recent deployment:
# - Check: Was there a deployment in last 1 hour?
# - If yes: Rollback immediately
kubectl rollout undo deployment/app  # Kubernetes
git revert HEAD && git push          # Simple revert
\`\`\`

**Option 2: Traffic Rerouting**
\`\`\`bash
# Route traffic away from failing instances
kubectl delete pod <failing-pod>  # K8s restarts pod
# Or manually drain and recreate
\`\`\`

**Option 3: Scale Up Resources**
\`\`\`bash
# If resource exhaustion:
kubectl scale deployment/app --replicas=6  # Double capacity
\`\`\`

**Option 4: Disable Failing Feature**
\`\`\`bash
# Feature flag to disable problematic feature
curl -