/systematic-debugging

name: systematic-debugging
description: "4-phase root cause debugging: understand bugs before fixing."
version: 1.1.0
author: Hermes Agent (adapted from obra/superpowers)
license: MIT
platforms: [linux, macos, windows]
metadata:
  hermes:
    tags: [debugging, troubleshooting, problem-solving, root-cause, investigation]
    related_skills: [test-driven-development, plan, subagent-driven-development]

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

The Feedback Loop Rule

The feedback loop is the debugging work. Before reading code to build a theory, create or identify a **tight** command that can go red on the user's exact symptom and green when the bug is fixed. A tight loop is fast, deterministic, agent-runnable, and specific enough to catch this bug — not merely "doesn't crash".

When a clean repro is hard, spend disproportionate effort building the loop. Guessing without a red-capable loop is the failure mode this skill exists to prevent.

When to Use

Use for ANY technical issue:

• Test failures
• Bugs in production
• Unexpected behavior
• Performance problems
• Build failures
• Integration issues

**Use this ESPECIALLY when:**

• Under time pressure (emergencies make guessing tempting)
• "Just one quick fix" seems obvious
• You've already tried multiple fixes
• Previous fix didn't work
• You don't fully understand the issue

**Don't skip when:**

• Issue seems simple (simple bugs have root causes too)
• You're in a hurry (rushing guarantees rework)
• Someone wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.

---

Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. Read Error Messages Carefully

• Don't skip past errors or warnings
• They often contain the exact solution
• Read stack traces completely
• Note line numbers, file paths, error codes

**Action:** Use `read_file` on the relevant source files. Use `search_files` to find the error string in the codebase.

2. Build a Tight Feedback Loop

• Can you trigger the user's exact symptom with one command?
• Does the command fail for this bug and only pass once the bug is fixed?
• Is it fast enough to run repeatedly?
• Is it deterministic? For flaky bugs, can you raise the reproduction rate high enough to debug?
• If not reproducible → gather more data, don't guess.

**Ways to construct a loop — try in roughly this order:**

1. **Failing test** at the seam that reaches the bug: unit, integration, or end-to-end. 2. **HTTP script / curl** against a running dev server. 3. **CLI invocation** with fixture input, diffing stdout/stderr against expected output. 4. **Headless browser script** (Playwright/Puppeteer) asserting on DOM, console, or network. 5. **Replay a captured trace**: HAR, request payload, event log, queue message, or webhook body. 6. **Throwaway harness** that boots the smallest useful slice of the system and calls the failing path. 7. **Property / fuzz loop** when the bug is intermittent wrong output over a broad input space. 8. **Bisection harness** suitable for `git bisect run` when the bug appeared between two known states. 9. **Differential loop** comparing old vs new version, two configs, two providers, or two datasets. 10. **Human-in-the-loop script** only as a last resort: script the human steps and capture their result so the loop stays structured.

**Tighten the loop once it exists:**

• Make it faster: cache setup, narrow scope, skip unrelated initialization.
• Make the signal sharper: assert the exact symptom, not generic success.
• Make it more deterministic: pin time, seed randomness, isolate filesystem, freeze network.

For non-deterministic bugs, the immediate goal is a higher reproduction rate, not perfection. Run the trigger 100x, parallelize, add stress, narrow timing windows, or inject sleeps. A 50% flake is debuggable; a 1% flake usually is not.

**Action:** Use the `terminal` tool to run the tight loop:

# Run a specific failing test
pytest tests/test_module.py::test_name -v

# Or run a scripted repro
python scripts/repro_bug.py

# Or run a high-repetition flaky repro
for i in {1..100}; do pytest tests/test_flake.py::test_name -q || break; done

3. Check Recent Changes

• What changed that could cause this?
• Git diff, recent commits
• New dependencies, config changes

**Action:**

# Recent commits
git log --oneline -10

# Uncommitted changes
git diff

# Changes in specific file
git log -p --follow src/problematic_file.py | head -100

4. Gather Evidence in Multi-Component Systems

**WHEN system has multiple components (API → service → database, CI → build → deploy):**

**BEFORE proposing fixes, add diagnostic instrumentation:**

For EACH component boundary:

• Log what data enters the component
• Log what data exits the component
• Verify environment/config propagation
• Check state at each layer

Run once to gather evidence showing WHERE it breaks. THEN analyze evidence to identify the failing component. THEN investigate that specific component.

5. Trace Data Flow

**WHEN error is deep in the call stack:**

• Where does the bad value originate?
• What called this function with the bad value?
• Keep tracing upstream until you find the source
• Fix at the source, not at the symptom

**Action:** Use `search_files` to trace references:

# Find where the function is called
search_files("function_name(", path="src/", file_glob="*.py")

# Find where the variable is set
search_files("variable_name\\s*=", path="src/", file_glob="*.py")

Phase 1 Completion