/observability

name: observability
description: This skill should be used when the user asks about "observability" or "monitoring", what "metrics, logs, and traces" to collect, "health checks" (liveness/readiness), "alerting" or "on-call", "SLO/SLI" or "error budgets", the "RED" or "USE" method, "dashboards", or names a tool like "Prometheus", "Grafana", or "Datadog". Use it whenever a design has no answer to "how would we know this is broken?" or "what do we alert on?" — i.e. any time failure would be invisible until users complain, even if the user doesn't say "observability".

Observability

Decide *what to measure* so a system can be seen, alerted on, and debugged in production. Getting this wrong is failure mode #6 — ignoring failure: a design that works on the whiteboard but goes dark under load, where the first signal of an outage is a user complaint instead of a page.

When to reach for this

Any production design needs an answer to "how would we know this broke, and how fast?" Reach for this when defining what the system measures, what pages a human, what an acceptable level of service is (SLO), or how a request is traced across services. It is the design move that makes every *other* block's stress section real — you cannot mitigate a thundering herd or a hot shard you can't see.

When NOT to

Do not build a full metrics-logs-traces stack for a prototype or an internal tool with no users to disappoint (YAGNI) — a health check and error logging are enough. Do not invent SLOs nobody will defend, or wire alerts before knowing the symptom that matters; an alert with no owner and no runbook is noise that trains the team to ignore pages. This skill owns *what* to measure and alert on; the **high-volume log pipeline** (collect → buffer → ship → index → retain) lives in `distributed-logging` — summarize and link, don't rebuild it here.

Clarify first

• **What is "healthy" from a user's view?** The symptom that defines a bad

experience (slow checkout, failed upload) — alerts target this, not CPU.

• **SLO target and window?** e.g. 99.9% of requests < 300 ms over 30 days. This

sets the error budget and the alert thresholds. (→ `back-of-the-envelope` for the nines.)

• **Request volume and cardinality?** QPS drives metric/trace sample rates;

high-cardinality labels (user ID, URL) blow up a metrics store.

• **Is this request-driven or resource-driven?** Picks RED (services) vs USE

(CPU/disk/queues) as the measurement frame.

• **Multi-service request path?** If a request crosses services, tracing earns

its keep; a single service may not need it yet.

The options

**The three pillars** (complementary, not either/or):

• **Metrics** — cheap numeric time series (counters, gauges, histograms). Use for

dashboards, trend analysis, and *alerting* — the always-on signal.

• **Logs** — discrete events with context. Use for debugging the specific failure

after an alert fires. (The pipeline that moves them is `distributed-logging`.)

• **Traces** — one request's journey across services with timing per hop. Use to

find *which* service or dependency is the latency/error source.

**What to measure** (pick a frame per component):

• **RED** (request-driven services): **R**ate, **E**rrors, **D**uration. Use for

APIs, web tiers, anything serving requests.

• **USE** (resources): **U**tilization, **S**aturation, **E**rrors. Use for CPU,

memory, disk, connection pools, queues.

• **Four golden signals** (latency, traffic, errors, saturation) — the superset;

use as the default service dashboard.

**Health checks** (the signal, consumed by `load-balancing`/orchestrator):

• **Liveness** — is the process alive / not deadlocked? Failure ⇒ *restart*. Keep

it cheap; don't check dependencies. Use to recover stuck processes.

• **Readiness** — can this instance serve traffic *now* (deps reachable, warmup

done)? Failure ⇒ *pull from rotation, don't restart*. Use to gate cold/struggling instances.

**Alerting**:

• **Symptom-based (SLO burn)** — page when the user-facing SLO is at risk. Use as

the default; it is actionable and low-noise.

• **Cause-based (resource thresholds)** — ticket/warn on CPU, disk, saturation.

Use for capacity planning, not paging.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | Metrics | Cheap, always-on alerting + trends | No per-request detail; high-cardinality labels explode storage/cost | You need to debug a *specific* request → add traces/logs | | Logs | Rich context for debugging an incident | Volume + cost; needs the `distributed-logging` pipeline to scale | Volume is unmanageable → sample, or shift detail to metrics | | Traces | Pinpoints the slow/failing hop across services | Instrumentation effort; sampling needed at high QPS | Single-service or low volume → defer; metrics suffice | | RED | Right frame for request services | Misses resource exhaustion that hasn't yet hurt requests | Component is a resource (queue, disk) → use USE | | USE | Catches saturation before it hurts users | Resource-centric, not user-centric; can be noisy | Alerting on it → switch to symptom/SLO-based | | Liveness check | Recovers deadlocked processes | Too aggressive ⇒ restart loops, mask real bugs | Restarts hide a crash-loop → fix readiness/root cause | | Readiness check | Keeps cold/broken instances out of rotation | Flapping if it checks flaky deps ⇒ capacity yo-yo | All instances fail readiness together → it's a dep outage | | SLO/error budget | Ties alerting + release pace to user pain | Effort to define + defend; wrong SLO misleads | Budget never burns (too loose) or always burns (too tight) | | Symptom alerting | Low-noise, actionable pages | Slightly slower to localize root cause | Need faster localization → add cause-based *tickets* (not pages) |

Behavior under stress

Observability is most needed exactly when it's most likely to break or mislead.

• **The monitoring amplifies the outage.** Synchronous logging on the reques