/observability-monitoring

name: observability-monitoring
description: "Design, audit, and troubleshoot production monitoring and observability using user-impact checks, layered telemetry, USE/RED, SLI/SLO/SLA, error budgets, cardinality controls, actionable alerting, burn-rate response, and postmortems. Use when asked about monitoring, наблюдаемость, алерты, Prometheus, Grafana, OpenTelemetry, logs, traces, profiles, service health, or incident evidence. Do not use for generic dashboard styling, frontend-only UI work, or unrelated code review."

Observability Monitoring

Use this skill to turn vague "is it working?" questions into evidence-backed monitoring, alerting, and incident workflows. Start from user or business impact, then move down through the system layers and choose the signal that can prove the current hypothesis.

Operating rule

Do not treat a green dashboard as proof of health. A monitoring claim is complete only when it names:

1. the observed scope and time window; 2. the user, business, or operator outcome being protected; 3. the signal and exact query/probe that supports the claim; 4. the threshold or SLO that defines bad; 5. the next human action and its runbook/evidence link.

Keep code review, live runtime proof, UI/render proof, and release readiness as separate verdicts.

Investigation is read-only by default. A restart, alert suppression, metric-schema/label change, sampling or retention change, or vendor reconfiguration is a production mutation: involve the responsible owner or incident authority, preserve the relevant evidence first, capture the exact config/command diff, state the rollback condition, and verify the user probe plus SLI after the change.

Workflow

1. Freeze scope and collect live facts

Before changing a monitor, alert, host, or service:

• Read the repository `AGENTS.md`, relevant rules, runbooks, and deployment docs.
• Establish the actual checkout/branch, deployment, process, host, port, proxy/tunnel, and data source.
• Record the observation time, environment, query/probe, and whether the result is current or historical.
• Inspect the source code/config that emits or consumes the signal before changing it.
• Never infer health from an old screenshot, a stale handoff, a single PID, or a dashboard with no successful user probe.

For infrastructure fixes, document the traffic path (DNS, proxy, tunnel, ingress, service, backend) before touching a surprising value such as `127.0.0.1`, a non-default port, or a disabled check.

2. Define the outcome before the metric

Write the protected outcome in concrete terms:

• **User:** can open the page, submit the request, download the result, or complete the workflow.
• **Business:** orders, successful jobs, conversions, revenue, or another domain event continue to occur.
• **Operator:** an on-call engineer receives a page early enough to act and can identify the next step.

Add at least one black-box or synthetic check for the user-facing path. Add real-user telemetry when the experience can vary by browser, geography, device, or network. Technical resource health is not a substitute for a user or business signal.

3. Map the monitored layers

Inspect the layers from bottom to top and state which ones are in scope:

1. **Hardware/infrastructure:** CPU, memory, disk capacity and latency, network, temperature, power, GPU, host availability. 2. **Host/OS:** load, processes, file descriptors, swap, service state, kernel/resource pressure. 3. **Network:** reachability, packet loss, latency, port state, DNS, TLS, ingress/proxy health. 4. **Application/APM:** request rate, errors, latency, saturation, dependency calls, queue depth. 5. **Databases and queues:** throughput, slow queries, connection pools, replication lag, backlog, consumer health. 6. **Containers/orchestration:** desired versus ready replicas, restarts, scheduling, evictions, resource limits, ephemeral identity. 7. **Business:** successful workflows, jobs, orders, revenue, conversion, or domain-specific zero-activity checks.

Do not stop at the infrastructure layer when the business outcome is failing. A system with green CPU and memory can still have a broken payment, form, queue consumer, or model job.

4. Choose the right signal

Use the signal that answers the question instead of collecting everything indiscriminately:

| Question | Primary signal | Practical method | |---|---|---| | Is a resource busy, queued, or failing? | Metrics | USE: utilization, saturation, errors | | Is a service serving users correctly? | Metrics | RED: rate, errors, duration | | What happened at a specific time? | Structured logs | Search by timestamp, service, severity, request/trace ID | | Where did a distributed request slow or fail? | Traces | Follow the trace across services and spans | | Why is code or a process consuming resources? | Profiles | Inspect sampled stacks, CPU, memory, locks, or I/O | | Do users actually succeed? | Synthetic/RUM/business probes | Run the workflow and inspect the domain result |

Metrics answer aggregate **how much/how often**. Logs answer **what happened**. Traces answer **where in the path**. Profiles answer **which code or process consumed the resource**. Correlate them with stable resource identity, timestamps, and trace/request context.

5. Design metrics without cardinality accidents

Treat every unique metric-name plus label set as a time series. Before adding a label, estimate its possible values and lifetime.

• Good metric dimensions are bounded and useful for aggregation: method, status class/code, service, region, queue, job type, or environment.
• Keep high-cardinality or per-event values in logs/traces: user ID, request ID, email, raw URL with parameters, exception text, or arbitrary object IDs.
• Prefer route templates over raw URLs.
• Set explicit cardinality limits or views where the telemetry SDK supports them, and monitor overflow/dropped-attribute indicators.
• Prefer histograms for fleet-wide aggreg