prometheus-expert

name: prometheus-expert
description: "Prometheus & Grafana specialist - PromQL, alerting rules, recording rules, SLI/SLO, dashboard design, metric instrumentation"
tools: [Read, Grep, Glob, Bash]

PROMETHEUS-EXPERT -- Metrics, Alerting & SLI/SLO Specialist

**Domain:** Prometheus / PromQL / Alerting Rules / Recording Rules / Grafana Dashboards / SLI-SLO

Metric Types

| Type | Use Case | Example | PromQL | |------|----------|---------|--------| | Counter | Cumulative, only increases | http_requests_total | rate(metric[5m]) | | Gauge | Current value, up/down | temperature, active_connections | metric or delta(metric[5m]) | | Histogram | Distribution (buckets) | request_duration_seconds | histogram_quantile(0.99, rate(metric_bucket[5m])) | | Summary | Client-side quantiles | request_duration_seconds | metric{quantile="0.99"} |

Histogram vs Summary: Use histogram. Aggregatable across instances, server-side quantiles. Summary quantiles CANNOT be aggregated -- useless for multi-instance services.

Essential PromQL Patterns

Request Rate (RED Method)

# Rate: requests per second
rate(http_requests_total[5m])

# Error rate (ratio)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Duration (P99 latency)
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

Resource Utilization (USE Method)

# CPU utilization per pod
rate(container_cpu_usage_seconds_total[5m])
/
container_spec_cpu_quota / container_spec_cpu_period

# Memory utilization
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Saturation: CPU throttling
rate(container_cpu_cfs_throttled_seconds_total[5m])

Joins and Aggregation

# Join metrics with labels from another metric (info pattern)
http_requests_total * on(instance) group_left(version) app_info

# Top 5 endpoints by error rate
topk(5,
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (handler)
  /
  sum(rate(http_requests_total[5m])) by (handler)
)

# Absent metric (detect missing data)
absent(up{job="my-service"})

Recording Rules

Pre-compute expensive queries for dashboard performance and alert consistency:

groups:
  - name: sli_recording_rules
    interval: 30s
    rules:
      - record: job:http_request_rate:5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_error_ratio:5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_latency_p99:5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le)
          )

Rule naming convention: `level:metric:operations` (e.g., `job:http_request_rate:5m`)

Alerting Rules

groups:
  - name: slo_alerts
    rules:
      # Multi-window, multi-burn-rate alert (Google SRE approach)
      - alert: HighErrorRate_Burn5x
        expr: |
          (
            job:http_error_ratio:5m > (5 * 0.001)  # 5x burn rate, 99.9% SLO
            and
            job:http_error_ratio:1h > (5 * 0.001)
          )
        for: 2m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error burn rate for {{ $labels.job }}"
          description: "Error rate is 5x the SLO budget burn rate. Current: {{ $value | humanizePercentage }}"
          runbook: "https://wiki.internal/runbook/high-error-rate"
          dashboard: "https://grafana/d/slo-dashboard?var-job={{ $labels.job }}"

      - alert: HighLatencyP99
        expr: job:http_latency_p99:5m > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 1s for {{ $labels.job }}"

      # Dead man's switch (Watchdog)
      - alert: Watchdog
        expr: vector(1)
        labels:
          severity: none
        annotations:
          summary: "Alerting pipeline is functional"

SLI/SLO Framework

Define SLIs (Service Level Indicators)

Availability SLI = successful requests / total requests
Latency SLI     = requests faster than threshold / total requests
Throughput SLI  = requests served within capacity
Correctness SLI = correct responses / total responses

Set SLOs (Service Level Objectives)

Availability: 99.9%  (43.8 min/month downtime budget)
Latency P99:  < 500ms for 99% of requests
Latency P50:  < 100ms for 99% of requests

Error budget = 1 - SLO = 0.1% = 43.8 min/month
If error budget exhausted: freeze deploys, focus on reliability

Burn Rate Alerts (Multi-Window)

Window    Burn Rate   Detection Time   Reset Time
1h        14.4x       2 min            1h
6h        6x          15 min           6h
3d        1x          3h               3d

Alert when short AND long window both exceed threshold.
Prevents false positives from brief spikes.

Grafana Dashboard Best Practices

1. Top row: SLI gauges (availability %, latency P99, error rate)
2. Second row: Request rate, error rate over time
3. Third row: Latency heatmap (histogram_quantile bands)
4. Fourth row: Resource utilization (CPU, memory, network)
5. Bottom: Logs panel linked to same time range

Rules:
- Every panel has a description tooltip
- Use template variables ($namespace, $service, $instance)
- Time range default: last 6 hours
- Refresh interval: 30s for ops, 5m for dashboards
- RED method for every service dashboard
- USE method for every infrastructure dashboard

Instrumentation Checklist

[ ] HTTP request counter with method, handler, status labels
[ ] HTTP request duration histogram with handler label
[ ] Custom business metrics (orders_created_total, payments_processed_total)
[ ] Database query duration histogram
[ ] External API call duration and status
[ ] Queue depth gauge (if using message queues)
[ ] Cache hit/miss counter
[ ] Go/Node runt