alerting-patterns

Alerting Patterns Reference

> **Scope**: SLO-based alerting, multi-window burn rate, and Alertmanager configuration patterns > **Version range**: Prometheus 2.0+ / Alertmanager 0.20+ > **Generated**: 2026-04-09 — verify burn rate math against your SLO targets

---

Overview

SLO-based alerting is the production standard for actionable alerts. The primary failure mode is alerting on symptoms (CPU, disk, memory) that have no direct user impact, or using single-window burn rate that misses both slow and fast burns. Google's SRE book multi-window burn rate pattern detects 99% of SLO violations with low false-positive rate.

---

SLO Burn Rate — Core Math

A burn rate of N means you're consuming your error budget N× faster than allowed.

| Burn Rate | Time to Exhaust Budget | Severity | Window | |-----------|----------------------|----------|--------| | 14.4× | 1 hour | Critical / Page | short: 1h, long: 5m | | 6× | 2.5 hours | Critical / Page | short: 6h, long: 30m | | 3× | 5 days | Warning / Ticket | short: 1d, long: 2h | | 1× | 30 days | No alert needed | — |

**Standard multi-window SLO alert (99.9% SLO, 30-day window)**:

groups:
  - name: slo_burn_rate
    rules:
      # Page immediately — exhausts budget in 1 hour
      - alert: SLOBurnRateCritical
        expr: |
          (
            job:slo_error_rate:ratio1h{job="api"} > (14.4 * 0.001)
            and
            job:slo_error_rate:ratio5m{job="api"} > (14.4 * 0.001)
          )
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "SLO burn rate critical: {{ $labels.job }}"
          description: "Error budget exhausted in < 1h at current rate"
          runbook_url: "https://wiki.example.com/runbooks/slo-burn-rate"

      # Ticket — exhausts budget in 2.5 hours
      - alert: SLOBurnRateHigh
        expr: |
          (
            job:slo_error_rate:ratio6h{job="api"} > (6 * 0.001)
            and
            job:slo_error_rate:ratio30m{job="api"} > (6 * 0.001)
          )
        for: 15m
        labels:
          severity: warning

The `0.001` is the error threshold for a 99.9% SLO (1 - 0.999). For 99.5% SLO, use `0.005`.

---

Correct Patterns

Alertmanager Inhibition Rules

Use inhibition to suppress lower-severity alerts when a higher-severity alert fires for the same service:

# alertmanager.yml
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal:
      - job
      - instance

**Why**: Without inhibition, a database outage fires both `DBDown (critical)` and `SlowQueries (warning)` for the same instance. On-call gets paged twice and must mentally correlate. Inhibition auto-suppresses the warning when the critical is active.

---

Alert Grouping by Team

Group alerts before routing to team channels:

# alertmanager.yml
route:
  group_by: [alertname, job, severity]
  group_wait: 30s      # wait 30s to batch alerts in same group
  group_interval: 5m   # send new alerts in existing group every 5m
  repeat_interval: 4h  # re-notify if still firing after 4h
  receiver: default

  routes:
    - match:
        team: platform
      receiver: platform-slack
      group_by: [alertname, cluster]

    - match:
        severity: critical
      receiver: pagerduty
      continue: true    # continue to check other routes too

**Why**: Without `group_by`, Alertmanager sends one notification per alert, generating 50+ Slack messages during an incident. Grouping collapses a K8s node failure (5+ firing alerts per pod) into one actionable message.

---

Pattern Catalog

<!-- no-pair-required: section header only -->

Use Multi-Window Burn Rate Alerts

**Detection**:

grep -rn 'burn_rate\|burnrate\|error_rate.*ratio' --include="*.yml" -A 5 | grep -v 'and'
rg 'alert.*[Bb]urn' --type yaml -A 8 | grep -v 'and\s*$'

**Signal**:

- alert: SLOBurnRate
  expr: job:error_rate:ratio5m > 0.01
  # Single window — misses slow burns over hours

**Why this matters**: A single 5-minute window catches fast burns (many errors quickly) but misses slow burns (few errors sustained over hours) that also exhaust the error budget. 30% of budget-exhausting incidents are slow burns invisible to single-window alerting.

**Preferred action**: Multi-window burn rate — require both a long window (confirming trend) and a short window (confirming it's ongoing):

- alert: SLOBurnRate
  expr: |
    job:error_rate:ratio1h > (14.4 * 0.001)
    and
    job:error_rate:ratio5m > (14.4 * 0.001)
  for: 2m

---

Validate Alertmanager Config with amtool Before Applying

**Detection**:

# Check if amtool is available and used in CI
grep -rn 'amtool' --include="Makefile" --include="*.sh" --include="*.yml"
# If no results: amtool validation is missing from deployment pipeline

<!-- no-pair-required: partial section — positive counterpart follows in next block -->

**Signal**:

# alertmanager.yml applied directly without validation
kubectl apply -f alertmanager-config.yaml
# A YAML syntax error silences ALL alerts

**Why this matters**: A single YAML syntax error in `alertmanager.yml` causes Alertmanager to reject the config and continue using the previous valid config — or fail to start. No error surfaces in the UI until alerts fail to route. Silent alert failures are worse than loud ones.

**Preferred action**:

# Validate before applying
amtool check-config alertmanager.yml

# Test routing for a specific alert label set
amtool config routes test --config.file=alertmanager.yml \
  severity=critical job=api team=platform

# In CI: add this to pre-commit or CI pipeline
amtool check-config alertmanager.yml && echo "Config valid"

---

Include Runbook Annotations on Every Alert

**Detection**:

grep -rn '^\s*- alert:' --include="*.yml" -A 15 | grep -B 10 'severity: critical' | grep -v 'runbook'
rg 'alert:' --type yaml -A 12 | grep -B8 'severity: crit