/monitoring-observability

name: monitoring-observability
description: "Monitoring, metrics, alerting, and observability standards. TRIGGER when: implementing health checks, collecting metrics, or defining alert rules. SKIP: log statement formatting (use logging-standards); CI pipeline setup (use github-actions-template)."
context: fork
agent: general-purpose

Monitoring & Observability Skill

Purpose

Standards for monitoring, metrics, alerting, and observability.

Auto-Invoke Triggers

• Setting up monitoring infrastructure
• Defining metrics and KPIs
• Configuring alerts
• Implementing distributed tracing

---

Three Pillars of Observability

| Pillar | Purpose | Tools | |--------|---------|-------| | **Logs** | Event records | ELK, Loki, CloudWatch | | **Metrics** | Numerical measurements | Prometheus, Datadog | | **Traces** | Request flow | Jaeger, Zipkin, X-Ray |

---

Key Metrics (Golden Signals)

The Four Golden Signals

| Signal | Description | Example Metric | |--------|-------------|----------------| | **Latency** | Response time | p50, p95, p99 latency | | **Traffic** | Request volume | Requests per second | | **Errors** | Failure rate | Error percentage | | **Saturation** | Resource usage | CPU, memory utilization |

RED Method (Request-focused)

• **R**ate - Requests per second
• **E**rrors - Failed requests per second
• **D**uration - Request latency

USE Method (Resource-focused)

• **U**tilization - Resource % used
• **S**aturation - Queue depth
• **E**rrors - Error count

---

Application Metrics

HTTP Endpoints

| Metric | Type | Description | |--------|------|-------------| | `http_requests_total` | Counter | Total requests | | `http_request_duration_seconds` | Histogram | Request latency | | `http_requests_in_flight` | Gauge | Active requests | | `http_response_size_bytes` | Histogram | Response size |

Database

| Metric | Type | Description | |--------|------|-------------| | `db_connections_active` | Gauge | Active connections | | `db_query_duration_seconds` | Histogram | Query time | | `db_errors_total` | Counter | Query errors |

Business Metrics

| Metric | Type | Description | |--------|------|-------------| | `users_registered_total` | Counter | New registrations | | `orders_created_total` | Counter | Orders placed | | `payment_amount_total` | Counter | Revenue |

---

Metric Types

| Type | Use Case | Example | |------|----------|---------| | **Counter** | Cumulative totals | Requests, errors | | **Gauge** | Current value | Temperature, queue size | | **Histogram** | Value distribution | Latency buckets | | **Summary** | Quantiles | p50, p95, p99 |

Naming Convention

{namespace}_{subsystem}_{name}_{unit}

http_server_request_duration_seconds
db_pool_connections_active
app_users_registered_total

---

Alerting Strategy

Alert Severity

| Severity | Response Time | Action | |----------|---------------|--------| | Critical | Immediate | Page on-call | | Warning | Within hours | Create ticket | | Info | Next business day | Review |

Alert Rules

• Alert on symptoms, not causes
• Include runbook links
• Set appropriate thresholds
• Avoid alert fatigue
• Group related alerts

What to Alert On

| Alert | Condition | Severity | |-------|-----------|----------| | Service down | Health check fails | Critical | | High error rate | > 5% errors | Critical | | High latency | p99 > 2s | Warning | | High CPU | > 80% for 5min | Warning | | Disk space | < 20% free | Warning | | SSL expiry | < 30 days | Warning |

---

SLOs and SLIs

Service Level Indicators (SLIs)

• Availability: Successful requests / Total requests
• Latency: % requests < threshold
• Throughput: Requests per second
• Error rate: Failed requests / Total requests

Service Level Objectives (SLOs)

| SLO | Target | Error Budget | |-----|--------|--------------| | Availability | 99.9% | 43.8 min/month | | Latency (p99) | < 500ms | - | | Error rate | < 0.1% | - |

Error Budget

• Monthly allowed downtime
• Spend on risky deployments
• Freeze deploys when exhausted

---

Distributed Tracing

Concepts

| Term | Description | |------|-------------| | Trace | End-to-end request journey | | Span | Single operation in trace | | Context | Trace ID propagated across services |

What to Trace

• Cross-service calls
• Database queries
• External API calls
• Message queue operations

Trace Propagation

• Pass trace context in headers
• Standard: W3C Trace Context
• Headers: `traceparent`, `tracestate`

---

Health Checks

Endpoint Types

| Endpoint | Purpose | Response | |----------|---------|----------| | `/health` | Basic liveness | 200 OK | | `/health/ready` | Full readiness | 200 + deps status | | `/health/live` | Process alive | 200 OK |

Readiness Check Components

• Database connection
• Cache connection
• External service connectivity
• Required configuration present

Health Response Format

{
  "status": "healthy",
  "checks": {
    "database": "healthy",
    "cache": "healthy",
    "external-api": "degraded"
  },
  "version": "1.2.3"
}

---

Dashboard Design

Layout Principles

• Most important metrics at top
• Group related metrics
• Use consistent time ranges
• Include context (deploy markers)

Standard Panels

1. **Overview** - Traffic, errors, latency 2. **Resources** - CPU, memory, disk 3. **Dependencies** - DB, cache, external APIs 4. **Business** - Domain-specific metrics

Best Practices

• Link to runbooks
• Add annotations for deploys
• Use consistent colors
• Set reasonable refresh rates

---

Runbooks

Structure

1. **Alert description** - What triggered 2. **Impact** - User/business effect 3. **Diagnosis steps** - How to investigate 4. **Resolution steps** - How to fix 5. **Escalation** - Who to contact

Required Runbooks

• Service restart procedure
• Database failover
• Rollback deployment
• Scale up/down
• Incident communication

---

Best Practices

DO

• Monitor the four golden signa