/sota-performance

name: sota-performance
description: >-
  State-of-the-art performance engineering for building fast systems and
  auditing existing code for bottlenecks. Use when the task involves
  performance, optimization, latency, profiling, slow code, memory usage,
  caching, or throughput — designing latency budgets, fixing N+1 and
  accidental-quadratic patterns, tuning allocation/GC pressure, network and
  I/O efficiency, cache architecture, Core Web Vitals, or setting up
  benchmarks and regression gates. Not for concurrency correctness (races,
  deadlocks, cancellation) — use sota-async-concurrency. Trigger keywords: performance,
  optimization, latency, profiling, slow, memory usage, caching, throughput,
  bottleneck, p99, flamegraph, Core Web Vitals.

SOTA Performance Engineering

Purpose

Make systems fast by default and find why they are slow by evidence. This skill encodes two disciplines that share one rule set:

1. **BUILD** — write code whose performance characteristics are known, budgeted, and protected by regression tests before it ships. 2. **AUDIT** — read existing code and telemetry to locate bottlenecks, rank them by user-facing impact, and prescribe fixes with expected gains.

Core doctrine: **measure first, but fix known pathologies on sight.** Profiling is mandatory before micro-optimization; it is NOT required to remove an O(n²) loop, an N+1 query, or an unbounded cache. "Premature optimization" never excuses shipping a known pathology.

BUILD mode

When writing new code or features:

1. **Set a budget before writing.** Define the latency budget (p99, not average) and decompose it across hops. An endpoint with a 200 ms p99 budget that calls auth (10 ms) + 2 DB queries (2×15 ms) + serialization (5 ms) has 155 ms of headroom — spend it consciously. See `rules/01-methodology.md`. 2. **Choose data structures by access pattern, not habit.** Know the n. n < 100: anything works. n unbounded: complexity class is the design. See `rules/02-algorithms-data-structures.md`. 3. **Batch and stream at every boundary.** One round trip per collection, not per item. Stream large results; never materialize unbounded data. 4. **Control allocation in hot paths.** Pre-size collections, reuse buffers, avoid per-iteration allocation in loops that run > 10⁴ times per second. See `rules/03-memory.md`. 5. **Make I/O cheap by construction.** Pooled connections, keep-alive, buffered writes, compression chosen per payload type. See `rules/04-io-network.md`. 6. **Cache deliberately or not at all.** Every cache ships with: key schema, TTL + jitter, eviction policy, invalidation path, stampede protection, and a hit-ratio metric. A cache missing any of these is a future incident. See `rules/05-caching.md`. 7. **Protect the win.** Add a benchmark or perf test in CI for any code with a budget. A perf improvement without a regression gate is a loan, not an asset. 8. **Frontend ships against Core Web Vitals budgets** (LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1 at p75). See `rules/06-frontend-web.md`.

AUDIT mode

How to find performance issues by reading code

Work outside-in, hottest path first:

1. **Identify the hot paths.** Entry points with highest traffic or strictest SLO: request handlers, queue consumers, render loops, cron jobs over large datasets. Audit those first; ignore cold admin paths until the end. 2. **Grep for pathology signatures** (high hit rate, low effort):

• Loops containing `await`/network/DB calls → N+1 (`rules/02`)
• String/array concatenation inside loops → accidental quadratic (`rules/02`)
• `.includes`/`in list`/linear `find` inside a loop → O(n·m) (`rules/02`)
• `SELECT *`, queries without LIMIT, missing pagination (`rules/02`, `rules/04`)
• Caches/maps with insert but no eviction or TTL → leak (`rules/03`, `rules/05`)
• `addEventListener`/subscribe without matching removal (`rules/03`)
• Sequential awaits on independent operations → serialized latency (`rules/04`)
• New client/connection per request instead of pooled (`rules/04`)
• Sync file/crypto/compression calls on async event loops (`rules/04`)
• `JSON.parse`/serialize of large payloads in hot loops (`rules/03`)

3. **Check the boundaries.** Most production latency lives at boundaries: process↔kernel (syscalls), service↔DB, service↔service, server↔browser. Count round trips per user action; > 3 sequential round trips is a finding. 4. **Check resource lifecycle.** Anything created per-request that is expensive to create (connections, TLS sessions, regexes, compiled templates, clients) should be created once and reused. 5. **Check what's missing**: no timeouts, no pagination, no backpressure, no pool bounds, no cache eviction — absent code is the most common perf bug.

What to measure (when you can run the system)

• **Latency distribution**: p50/p95/p99 per endpoint — never averages

(`rules/01`). Compare p99 to p50; ratio > 10× means contention, GC, or stampedes, not slow code.

• **USE per resource** (Utilization, Saturation, Errors): CPU, memory, disk,

network, pools, queues. **RED per service** (Rate, Errors, Duration).

• **Where time goes**: CPU flamegraph for compute, off-CPU/wall profile for

waiting. A request that is slow with idle CPU is blocked on I/O or locks.

• **Allocation rate and GC pause time** for managed runtimes.
• **Cache hit ratios** and **DB round trips per request**.
• **Frontend**: field CWV (CrUX/RUM) at p75, not lab-only Lighthouse.

Severity conventions (by user-facing impact)

| Severity | Criteria | |---|---| | **Critical** | Active or imminent user-facing failure: unbounded growth (memory leak, unpaginated scan) that will OOM/timeout at production scale; O(n²)+ on user-controlled input; stampede-capable cache in front of a fragile origin; p99 SLO breached now. | | **High** | Measurable user-facing degradation: N+1 on a hot path; missing pool/keep-alive adding RTTs per reque