/systematic-debugging

name: systematic-debugging
description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
argument-hint: "<error-message-or-symptom>"

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

• Test failures
• Bugs in production
• Unexpected behavior
• Performance problems
• Build failures
• Integration issues

**Use this ESPECIALLY when:**

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

**Don't skip when:**

• Issue seems simple (simple bugs have root causes too)
• You're in a hurry (rushing guarantees rework)
• Manager wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. **Read Error Messages Carefully**

• Don't skip past errors or warnings
• They often contain the exact solution
• Read stack traces completely
• Note line numbers, file paths, error codes

2. **Reproduce Consistently**

• Can you trigger it reliably?
• What are the exact steps?
• Does it happen every time?
• If not reproducible → gather more data, don't guess

3. **Check Recent Changes**

• What changed that could cause this?
• Git diff, recent commits
• New dependencies, config changes
• Environmental differences

4. **Gather Evidence in Multi-Component Systems**

**WHEN system has multiple components (CI → build → signing, API → service → database):**

**BEFORE proposing fixes, add diagnostic instrumentation:**

   For EACH component boundary:
     - Log what data enters component
     - Log what data exits component
     - Verify environment/config propagation
     - Check state at each layer

   Run once to gather evidence showing WHERE it breaks
   THEN analyze evidence to identify failing component
   THEN investigate that specific component

**Example (multi-layer system):**

   # Layer 1: Workflow
   echo "=== Secrets available in workflow: ==="
   echo "IDENTITY: ${IDENTITY:+SET}${IDENTITY:-UNSET}"

   # Layer 2: Build script
   echo "=== Env vars in build script: ==="
   env | grep IDENTITY || echo "IDENTITY not in environment"

   # Layer 3: Signing script
   echo "=== Keychain state: ==="
   security list-keychains
   security find-identity -v

   # Layer 4: Actual signing
   codesign --sign "$IDENTITY" --verbose=4 "$APP"

**This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)

5. **Trace Data Flow**

**WHEN error is deep in call stack:**

See `root-cause-tracing.md` in this directory for the complete backward tracing technique.

**Quick version:**

• Where does bad value originate?
• What called this with bad value?
• Keep tracing up until you find the source
• Fix at source, not at symptom

Phase 2: Pattern Analysis

**Find the pattern before fixing:**

1. **Find Working Examples**

• Locate similar working code in same codebase
• What works that's similar to what's broken?

2. **Compare Against References**

• If implementing pattern, read reference implementation COMPLETELY
• Don't skim - read every line
• Understand the pattern fully before applying

3. **Identify Differences**

• What's different between working and broken?
• List every difference, however small
• Don't assume "that can't matter"

4. **Understand Dependencies**

• What other components does this need?
• What settings, config, environment?
• What assumptions does it make?

Phase 3: Hypothesis and Testing

**Scientific method:**

1. **Form Single Hypothesis**

• State clearly: "I think X is the root cause because Y"
• Write it down
• Be specific, not vague

2. **Test Minimally**

• Make the SMALLEST possible change to test hypothesis
• One variable at a time
• Don't fix multiple things at once

3. **Verify Before Continuing**

• Did it work? Yes → Phase 4
• Didn't work? Form NEW hypothesis
• DON'T add more fixes on top

4. **When You Don't Know**

• Say "I don't understand X"
• Don't pretend to know
• Ask for help
• Research more

Phase 4: Implementation

**Fix the root cause, not the symptom:**

1. **Create Failing Test Case**

• Simplest possible reproduction
• Automated test if possible
• One-off test script if no framework
• MUST have before fixing
• Use the `ultraship:test-driven-development` skill for writing proper failing tests

2. **Implement Single Fix**

• Address the root cause identified
• ONE change at a time
• No "while I'm here" improvements
• No bundled refactoring

3. **Verify Fix**

• Test passes now?
• No other tests broken?
• Issue actually resolved?

4. **If Fix Doesn't Work**

• STOP
• Count: How many fixes have you tried?
• If < 3: Return to Phase 1, re-analyze with new information
• **If ≥ 3: STOP and question the architecture (step 5 below)**
• DON'T attempt Fix #4 without architectural discussion

5. **If 3+ Fixes Failed: Question Architecture**

**Pattern indicating architectural problem:**

• Each fix reveals new shared state/coupling/problem in different place
• Fixes require "massive refactoring" to implement
• Each fix creates new symptoms elsewhere

**STOP and question fundamentals:**

• Is this pattern fundamentally sound?
• Are we "sticking with it through sheer inertia"?
• Sh