/performance-profiling

name: performance-profiling
description: "Performance: golden signals, p50/p95/p99, flame graphs, load testing. Triggers: performance, slow, latency, p99, flame graph, bottleneck, memory leak."
effort: medium
allowed-tools: Read, Grep
user-invocable: false

Performance Profiling Skill

Optimization Golden Rule

**"Don't optimize without a baseline."** Always measure -> change -> measure.

Critical Metrics (The 4 Golden Signals)

1. **Latency**: Time it takes to serve a request. (p50, p95, p99) 2. **Traffic**: Demand on your system (req/sec). 3. **Errors**: Rate of requests that fail. 4. **Saturation**: How "full" your service is (CPU/Memory usage).

Profiling Tools & Techniques

Python

• **CPU Sampling**: `py-spy`

  # Record flamegraph
  py-spy record -o profile.svg --pid <pid>

• **Function Profiling**: `cProfile`

  import cProfile
  cProfile.run('main()')

Node.js

• **Flamegraphs**: `0x` or built-in profiler.

  node --prof app.js
  node --prof-process isolate-0xnnnnn.log > processed.txt

• **Event Loop**: `clinic doctor`

Database (SQL)

• **Explain Plan**: Analyze query cost.

  EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM users WHERE active = 1;

• **N+1 Problem**: Look for loop-generated queries.

Frontend (Browser)

• **Lighthouse**: Core Web Vitals (LCP, CLS, INP).
• **Chrome Performance Tab**: Main thread blocking time.
• **Network Waterfall**: Time to First Byte (TTFB).

Optimization Hierarchy

1. **Database/IO**: (Indexing, Caching, Batching) - *Biggest Gains* 2. **Algorithm**: (O(n²) -> O(n log n)) 3. **Memory**: (Allocation churn, GC pressure) 4. **Micro-optimization**: (Loop unrolling, etc.) - *Smallest Gains*

Common Rationalizations

| Excuse | Why It's Wrong | |--------|----------------| | "It feels slow, let me optimize this function" | Feelings aren't data — profile first, then optimize the actual bottleneck | | "We should optimize everything" | Premature optimization is the root of all evil — focus on the critical path | | "Caching will fix it" | Caching masks problems and adds complexity — fix the root cause first | | "It's fast enough in dev" | Dev has 1 user — production has thousands and cold caches | | "We'll optimize later" | Performance debt compounds — a 100ms regression per sprint = 5s in a year |

Example

# Capture a 30-second CPU flamegraph from a running Python service
py-spy record -o profile.svg --duration 30 --pid "$(pgrep -f my-service)"

# Identify top 3 hot functions
py-spy top --pid "$(pgrep -f my-service)"

Then, per the optimization hierarchy, start with DB/IO fixes (indexing, batching, caching the right layer) before touching algorithm-level changes.

Rules

• **MUST** capture a baseline measurement before proposing any change
• **NEVER** optimize code without profiler data pointing at it as the bottleneck
• **CRITICAL**: report p95/p99, not just p50 — averages hide real user pain
• **MANDATORY**: follow the hierarchy — DB/IO before algorithm before micro-optimization

Gotchas

• `py-spy` needs `CAP_SYS_PTRACE` on Linux and SIP-disabled codesigning on macOS to attach to another process. Containerized services usually run without ptrace privileges — profiling requires a `--cap-add=SYS_PTRACE` on the container or an in-process alternative (`cProfile`, `yappi`).
• Production hosts frequently set `/proc/sys/kernel/perf_event_paranoid=2` or higher, which disables user-space perf events. Tools that rely on perf (`perf`, `bcc`, `bpftrace`) silently produce empty output — check `cat /proc/sys/kernel/perf_event_paranoid` first.
• Node.js `--prof` output gets interleaved across worker threads and child processes. A single `isolate-*.log` mixes samples from multiple isolates unless each worker writes its own — filter by PID or use `clinic flame` which handles the split.
• Chrome DevTools samples at ~1kHz; operations faster than ~1ms vanish. For microbenchmarks, prefer `performance.now()` with manual markers, not the Performance tab.
• `EXPLAIN ANALYZE` on Postgres **executes** the query, including `INSERT`/`UPDATE`/`DELETE` — wrap write queries in a transaction that you roll back, or use `EXPLAIN (ANALYZE, BUFFERS) ... ; ROLLBACK;` in one statement.

When NOT to Use

• For correctness bugs (wrong output) — use `/debug`
• For frontend render bugs without timing data — measure with DevTools first
• For infrastructure capacity planning — use load testing, not profiling
• For generic code quality — use `/analyze`