/observability

name: observability
description: 'Use when adding telemetry, composing an observability surface, reviewing alerts, shipping a production feature, or diagnosing a production issue. Not for live failures, profiling, or launch runbooks.'

Observability

Contract

| Field | Bound contract | |---|---| | Trigger | Adding telemetry, composing the smallest observability surface the system will keep, reviewing alerting rules, shipping a production feature, or diagnosing an opaque production issue. | | Authority | Reversible local: writes only instrumentation code and local telemetry configuration in the working tree; rollback is discarding the edits. No remote mutation. | | Side effect | Adds instrumentation: structured log calls, metric instruments, tracer setup, alert definitions, and dashboard panels to the target code. | | Done | Structured logs carry a correlation ID, RED metrics exist with bounded labels, one request traces end-to-end with minimum spans and no broken spans, every dashboard panel maps to a failure mode, no duplicate or unowned surface remains, the surface compiles, loads, and emits locally, symptom-based alerts are defined, and local telemetry emission is verified. |

Inputs

Required: the target source files or endpoints to instrument, and their runtime (language and framework). Optional: existing logging, metrics, or tracing libraries; a pinned metrics or tracing backend; SLOs or historical latency and error data for threshold justification; the alert delivery channel and runbook location. When no backend is pinned, write against the vendor-neutral OpenTelemetry APIs so the exporter can be configured later.

Procedure

1. Read the named target code and confirm its runtime and write surface. If the target cannot be identified, stop without writing. Done when: the target code is read and its runtime and write surface are confirmed. 2. Write down 2-4 on-call questions for the feature (for example: what fraction of attempts succeed on the first try; why does a permanent failure happen; is the provider slower than usual). Every signal added below must answer one of these questions and map to a decision an operator will act on; reject any signal that does not. If no question can be named, stop and report; do not instrument. Done when: 2-4 on-call questions are written, each mapping to at least one signal below, and every signal maps to an operator action. 3. Map each question to one signal: how often or how fast in aggregate, a metric; where time goes across services, a trace; what happened in one specific case, a log. Instrument RED (rate, errors, duration) on every request-driven endpoint and external dependency; instrument USE (utilization, saturation, errors) on queues, pools, and hosts. Done when: each question is mapped to a signal type with RED/USE coverage applied. 4. Add structured logging: every line is a JSON object with a stable event name and machine-readable fields (IDs, provider, error code, attempt count). Never interpolate values into prose strings. Use levels consistently: `error` for broken invariants needing investigation, `warn` for degraded but handled, `info` for significant business events, `debug` off in production by default. Done when: structured JSON logging is added with consistent levels and no prose interpolation. 5. Generate or accept a request ID at the system boundary (for example the `x-request-id` header, else a UUID), attach it to every log line, span, and outbound call, and echo it on the response. Without it a single request cannot be reconstructed from interleaved logs. Done when: a request ID is generated at the boundary and propagated to every log line, span, and outbound call. 6. Never log secrets, tokens, passwords, or unredacted PII. Allowlist logged fields; never log whole request bodies. Done when: secret and PII logging is prevented with an allowlist. 7. Add metrics: a latency histogram per endpoint and dependency (for example `http_request_duration_seconds`, buckets spanning roughly 0.05s to 5s, labels `method`, route template, and `status_class` holding `2xx`/`5xx` classes, never the raw status code). Read p50/p95/p99, never averages. Labels come only from small fixed sets (route template, status class, provider name); never user IDs, emails, request IDs, full URLs, or error message text. Done when: latency histograms with bounded labels are added per endpoint and dependency. 8. Add tracing: enable OpenTelemetry auto-instrumentation for HTTP, gRPC, and database clients, initialized before application code, with the service name set. Add manual spans only around meaningful internal units of work, carrying the attributes on-call will filter by. Keep span count to the minimum that reconstructs the critical request path. Propagate context across every async boundary (HTTP headers, queue message metadata) or the trace dies at the gap. Sample head-based at a low rate; keep all errors via tail sampling when the backend supports it. Done when: tracing is enabled with auto-instrumentation, minimum manual spans for the critical path, and context propagation across async boundaries. 9. Define symptom-based alerts on what users feel: sustained error rate over a small percentage, p99 latency over seconds, queue age over minutes. Never alert on causes like CPU, pod restarts, or disk usage. Each alert must be actionable (if the response is to ignore it, delete it), link a runbook stating its meaning, first query, and escalation path, carry a threshold and duration justified by the SLO or historical data, and use exactly two severities: `page` (user-facing, act now) and `ticket` (degradation, act this week). Done when: symptom-based alerts are defined with runbook links, justified thresholds, and exactly two severities. 10. Build the dashboard around the failure modes revealed by the chosen signals and traces. Every panel must map to a failure mode or an on-call question; omit charts no one will read. Done when: every dashboar