/rescue

name: rescue
description: "Production Incident Commander — diagnose and recover from production incidents. Use when something is broken in production, site is down, errors spiking, or user reports a critical bug."
argument-hint: "<url-or-error-description>"

Production Incident Commander

When production is down, every minute costs trust. This skill runs an incident like a principal SRE — fast triage, clear decision-making, structured recovery, and prevention so it never happens again.

Severity Classification

Before doing anything, classify the incident:

| Severity | Definition | Response Time | Example | |---|---|---|---| | **SEV-1** | Service completely down, all users affected | Immediately | Site returns 500, database unreachable | | **SEV-2** | Major feature broken, many users affected | Within 15 min | Auth broken, payments failing, data loss | | **SEV-3** | Minor feature broken, some users affected | Within 1 hour | One API endpoint slow, email not sending | | **SEV-4** | Cosmetic or edge case | Next business day | UI glitch on one browser, non-critical error log |

Severity determines urgency. SEV-1/2: restore first, investigate later. SEV-3/4: investigate first, then fix.

Process

Phase 1: Gather Context

Ask for: 1. **Production URL** (if not already known) 2. **What's happening?** (down, slow, errors, specific feature broken) 3. **When did it start?** (narrows the commit search window) 4. **What changed recently?** (deploy, config change, dependency update, traffic spike)

If the user is panicking, skip questions and use whatever info is available. Speed > completeness for SEV-1.

Phase 2: Run Diagnostics

node ${CLAUDE_PLUGIN_ROOT}/tools/incident-commander.mjs <project-directory> --url=<production-url>

Parse the JSON output.

**If a Sentry MCP server is connected** (check your available tools — search for `sentry` tools), pull live production errors before guessing at code: list the most recent / most frequent issues since the incident window, read the top stack traces, and map each frame back to a file and line in this repo. A real stack trace from production beats inferring the culprit from recent commits. Use the actual error signature to narrow the suspect commit. If no Sentry server is connected, continue with the static diagnostics above (and mention that connecting Sentry would sharpen this step).

Phase 3: Triage

Present findings in order of urgency:

**Site Status:**

• UP / DOWN / DEGRADED
• Response time and status code
• Health endpoint status
• SSL certificate validity

**Likely Culprit:**

• Most recent commit with significant changes
• Files changed in that commit
• When it was deployed
• Correlation: did the issue start after this deploy?

**Error Patterns Found:**

• Unhandled promises, missing error handlers
• Environment variable issues (missing, placeholder values)
• Database connection problems
• Third-party service failures

**Resource Issues:**

• Memory pressure signals (process.memoryUsage patterns in code)
• Unbounded data growth (arrays that grow without cleanup)
• Connection pool exhaustion (too many DB connections)

Phase 4: Recovery Options

Present in order of speed — for SEV-1/2, always recommend Option 1 first:

**Option 1: Rollback (fastest — 2-5 min)**

git revert <culprit-hash> --no-edit && git push

This is almost always the right first move. Restore service, then investigate.

**When NOT to rollback:**

• The rollback would cause data loss (destructive migration already ran)
• The issue isn't in the latest deploy (pre-existing problem that suddenly surfaced)
• The rollback is bigger than the fix (e.g., reverting 50 files when the fix is 1 line)

**Option 2: Hot Fix (5-15 min)** If the error pattern is clear and the fix is small:

• Apply the fix using Edit tool
• Run tests locally
• Push the fix with a clear commit message: `fix: [what was broken] — incident [date]`
• Verify with health check

**Option 3: Traffic Management (immediate)** If the issue is load-related:

• Enable maintenance mode if available
• Scale up infrastructure if possible (Railway: increase instance count)
• Add rate limiting to affected endpoints
• Redirect traffic away from broken feature

**Option 4: Investigate Further** If the cause isn't clear:

• Check application logs (Railway: `railway logs`, Vercel: function logs)
• Check database connectivity and query performance
• Check third-party service status pages (Stripe, Resend, Supabase, etc.)
• Check recent environment variable changes
• Check if DNS/SSL certificate expired

Phase 5: Verify Recovery

After applying a fix:

node ${CLAUDE_PLUGIN_ROOT}/tools/health-check.mjs <production-url>

Confirm the site is back to healthy status. Check:

• Status code 200
• Response time within normal range
• SSL still valid
• Key functionality working (not just the homepage)

Phase 6: Communication

For SEV-1/2, the user needs to communicate with their users:

**Status page update template:**

[Investigating] We're aware of [issue description] and are actively working on a fix.
[Identified] We've identified the cause and are deploying a fix.
[Resolved] The issue has been resolved. [Brief explanation]. We apologize for the disruption.

**If the user has a status page:** help them post the update. **If they don't:** suggest setting up a simple one (Instatus, Betteruptime, or a static page).

Phase 7: Post-Mortem

Generate a post-mortem document from the incident-commander output:

# Incident Post-Mortem — [Date]

## Summary
- **What happened:** [One sentence]
- **Severity:** SEV-[N]
- **Duration:** [start time] to [end time] ([N] minutes)
- **Impact:** [Who was affected, what they experienced]
- **Root cause:** [One sentence]

## Timeline
| Time | Event |
|---|---|
| HH:MM | Issue detected (how: monitoring/user report/deploy) |
| HH:MM | Investigation started |
| HH:MM | Root cause identified |
| HH:MM | Fix deployed |
| HH:MM | Servic