/incident-response

description: Coordinate P0 incident response — hotfix validation, timeline reconstruction, and post-incident analysis
argument-hint: "[issue number | \"active\" | \"postmortem {issue}\"]"
install: extras

<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios --> <!-- SPDX-License-Identifier: AGPL-3.0-or-later -->

/incident-response — P0 Incident Coordination

> **Skill names**: `{FORGE_SKILL_PREFIX}` is `forgedock:` (plugin install) or empty (`install.sh`), resolved once per run by `commands/work-on.md` § Skill Name Resolution. If the skill is not found under either name, STOP and report "skill not found" — never run the phase inline.

**Input**: $ARGUMENTS

**Config variables used by this command** (set in `forge.yaml`):

• `{REPO_PATH}` ← `paths.root` — project repository root
• `{DEPLOY_WORKFLOW}` ← `deploy.workflow` (optional) — GitHub Actions workflow filename for hotfix deploys (e.g., `hotfix-deploy.yml`). When absent or empty, workflow-trigger and workflow-monitor steps are omitted.

You are the pipeline's incident response coordinator. When production goes down or a critical bug surfaces, this command orchestrates the response: validates the hotfix before deploy, reconstructs the incident timeline, and produces a post-incident analysis.

**Agent model policy**: `model: "{DEFAULT_MODEL}"` — resolved from forge.yaml `agents.default_model`, else "sonnet" (standard tier). Fallback: `model: "opus"` if rate-limited. User can override with `--model <name>`. Feature gate: pass `effort` in Task/Skill spawns only on Claude Code >= 2.1.154.

**NEVER use plan mode (EnterPlanMode)** — it breaks execution context.

---

Command Router

| Input | Mode | Description | |-------|------|-------------| | Issue number (e.g., `55`) | **Active Response** | Coordinate hotfix for this P0 issue | | `active` or empty | **Active Response** | Find the most recent P0 issue and coordinate | | `postmortem {issue}` | **Post-Incident** | Generate postmortem for a resolved incident |

---

Mode 1: Active Incident Response

Phase 1: Identify the Incident

Step 1A: Find the P0 issue

# If issue number provided
gh issue view {NUMBER} --json number,title,body,labels,state,comments,assignees,createdAt

# If "active" — find most recent open P0
gh issue list --state open --label "priority:P0" --limit 5 --json number,title,createdAt,labels \
  --jq 'sort_by(.createdAt) | reverse | .[0]'

Step 1B: Establish incident context

Parse the issue to understand:

• **What's broken**: Which service, which endpoint, what user-facing impact
• **When it started**: Issue creation time, or mentioned timestamp
• **Suspected cause**: Recent deploy? Code change? External dependency?

# Check recent deploys (merges to main in last 24h)
gh pr list --state merged --base main --json number,title,mergedAt \
  --jq '[.[] | select(.mergedAt > (now - 86400 | todate))] | .[] | "\(.number) | \(.title) | \(.mergedAt)"'

# Check recent hotfix deploys (requires deploy.workflow set in forge.yaml)
# Skip this check if deploy.workflow is not configured.
gh run list --workflow={deploy.workflow} --limit 5 --json databaseId,status,conclusion,createdAt,displayTitle

Step 1C: Post incident acknowledgment

gh issue comment {NUMBER} --body "$(cat <<'EOF'
## 🚨 Incident Response Activated

**Coordinator**: Forge pipeline (automated)
**Time**: $(date -u +%Y-%m-%dT%H:%M:%SZ)

**Status**: Investigating
**Next action**: Validating fix approach and preparing hotfix deploy

---
### Timeline
| Time | Event |
|------|-------|
| {issue_created} | Incident reported |
| $(date -u +%H:%M) UTC | Response activated |
EOF
)"

---

Phase 2: Validate the Fix

If a fix branch already exists (check issue comments for PR references):

Step 2A: Find the fix

# Check if there's already a PR or branch
FIX_PR=$(gh pr list --state open --search "#{NUMBER}" --json number,headRefName --jq '.[0]')
FIX_BRANCH=$(echo $FIX_PR | jq -r '.headRefName // empty')

# Or check if issue comments mention a branch
gh api repos/{owner}/{repo}/issues/{NUMBER}/comments \
  --jq '.[].body | select(contains("fix/") or contains("hotfix/"))' | grep -oP '(fix|hotfix)/[\w-]+'

Step 2B: Validate the fix won't make things worse

Spawn a validation agent to check the proposed fix:

Agent(model="{SUBAGENT_MODEL}", prompt="
You are validating a hotfix for a P0 production incident.

Issue: #{NUMBER} — {TITLE}
Fix branch: {FIX_BRANCH}
Repo: {REPO_PATH}

Your job:
1. Read the fix diff: git diff origin/main..origin/{FIX_BRANCH}
2. Verify it addresses the reported problem (read the issue body)
3. Check for obvious regressions:
   - Does it break any imports?
   - Does it remove functionality other code depends on?
   - Are there syntax errors?
   - Does it change DB schema without a migration?
4. Run compile checks (read forge.yaml → verification.commands for tool commands):
   - Python: always run python -m py_compile on changed .py files (no config needed)
   - Python format/lint: run verification.commands.python.format and .lint if configured; log "SKIPPED — not configured in verification.commands" if absent
   - TypeScript: run verification.commands.typescript.typecheck if configured; log "SKIPPED — not configured in verification.commands" if absent
5. Check if the fix is scoped tightly (minimal changes for the incident)

Output:
- SAFE TO DEPLOY: {yes/no}
- Concerns: {list any non-blocking concerns}
- Confidence: {high/medium/low}
- Recommendation: {deploy immediately / needs adjustment / needs more investigation}
")

Step 2C: Report validation result

gh issue comment {NUMBER} --body "$(cat <<'EOF'
### Hotfix Validation

**Branch**: `{FIX_BRANCH}`
**Validation**: {SAFE TO DEPLOY / NEEDS ADJUSTMENT}
**Confidence**: {level}

{If safe}:
**Ready for deploy.** Run (requires `deploy.workflow` set in `forge.yaml`):

gh workflow run {deploy.workflow} --ref {FIX_BRANCH} -f {deploy.workflow_inputs.s