/systematic-debugging

name: systematic-debugging
description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

Systematic Debugging

Overview

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

• Test failures
• Bugs in production
• Unexpected behavior
• Performance problems
• Build failures
• Integration issues

**Use this ESPECIALLY when:**

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

**Don't skip when:**

• Issue seems simple (simple bugs have root causes too)
• You're in a hurry (rushing guarantees rework)
• Manager wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. **Read Error Messages Carefully**

• Don't skip past errors or warnings
• They often contain the exact solution
• Read stack traces completely
• Note line numbers, file paths, error codes

2. **Reproduce Consistently**

• Can you trigger it reliably?
• What are the exact steps?
• Does it happen every time?
• If not reproducible → gather more data, don't guess

3. **Check Recent Changes**

• What changed that could cause this?
• Git diff, recent commits
• New dependencies, config changes
• Environmental differences
• For workflow-level issues (blocked beads, stuck execution): `bd ready --explain` shows dependency reasoning
• Check the knowledge store for this symptom before investigating further: `bd memories <symptom-keywords>` (prior root-cause/lesson memories) and `bd list --label <topic> --status all` (+ `bd search "<symptom>" --status all` for decision/design knowledge-beads; error strings are body terms: add `--desc-contains "<error-string>"`; >10 hits: narrow the query, never triage truncated titles). Then read — hits are pointers, not knowledge: `bd show <ids>` every plausibly-matching prior decision/design, and `bd recall <key>` every plausibly-matching memory (`bd memories` prints truncated previews) — full bodies, before investigating further. 0 relevant does not mean none exist — re-angle the query once before emitting `KB check: none`. Emit `KB check: N hits, M read` + a one-line disposition each. If a prior root cause or decision already covers this, use it — don't re-debug something already understood.

4. **Gather Evidence in Multi-Component Systems**

**WHEN system has multiple components (CI → build → signing, API → service → database):**

**BEFORE proposing fixes, add diagnostic instrumentation:**

   For EACH component boundary:
     - Log what data enters component
     - Log what data exits component
     - Verify environment/config propagation
     - Check state at each layer

   Run once to gather evidence showing WHERE it breaks
   THEN analyze evidence to identify failing component
   THEN investigate that specific component

**Example (multi-layer system):**

   # Layer 1: Workflow
   echo "=== Secrets available in workflow: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

   # Layer 2: Build script
   echo "=== Env vars in build script: ==="
   env | grep IDENTITY || echo "IDENTITY not in environment"

   # Layer 3: Signing script
   echo "=== Keychain state: ==="
   security list-keychains
   security find-identity -v

   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"

**This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)

5. **Trace Data Flow**

**WHEN error is deep in call stack:**

See `root-cause-tracing.md` in this directory for the complete backward tracing technique.

**Quick version:**

• Where does bad value originate?
• What called this with bad value?
• Keep tracing up until you find the source
• Fix at source, not at symptom

Phase 2: Pattern Analysis

**Find the pattern before fixing:**

1. **Find Working Examples**

• Locate similar working code in same codebase
• What works that's similar to what's broken?

2. **Compare Against References**

• If implementing pattern, read reference implementation COMPLETELY
• Don't skim - read every line
• Understand the pattern fully before applying

3. **Identify Differences**

• What's different between working and broken?
• List every difference, however small
• Don't assume "that can't matter"

4. **Understand Dependencies**

• What other components does this need?
• What settings, config, environment?
• What assumptions does it make?

Phase 3: Hypothesis and Testing

**Scientific method:**

1. **Form Single Hypothesis**

• State clearly: "I think X is the root cause because Y"
• Write it down
• Be specific, not vague

2. **Test Minimally**

• Make the SMALLEST possible change to test hypothesis
• One variable at a time
• Don't fix multiple things at once

3. **Verify Before Continuing**

• Did it work? Yes → Phase 4
• Didn't work? Form NEW hypothesis
• DON'T add more fixes on top

4. **When You Don't Know**

• Say "I don't understand X"
• Don't pretend to know
• Ask for help
• Research more

Phase 4: Implementation

**Fix the root cause, not the symptom:**

1. **Create Failing Test Case**

• Simplest possible reproduction
• Automated test if possible
• One-off test script if no framework
• MUST have before fixing
• Use the `beads-superpowers:test-driven-development` skill for writing proper failing tests

2. **Implement Single Fix**

• Addre