promql-patterns

PromQL Patterns Reference

> **Scope**: PromQL query correctness, common expression mistakes, and recording rule design > **Version range**: Prometheus 2.0+ (most patterns apply to all 2.x) > **Generated**: 2026-04-09 — verify against current Prometheus release notes

---

Overview

PromQL is a functional query language where small mistakes produce silently wrong results rather than errors. The most common failure mode is using `rate()` or `increase()` incorrectly — queries return numbers that look plausible but are mathematically wrong. Detection requires knowing what correct output looks like, not just that the query runs.

---

Pattern Table

| Function | Version | Use When | Avoid When | |----------|---------|----------|------------| | `rate()` | 2.0+ | Sustained per-second rate over a window | Short windows (< 4x scrape interval) | | `irate()` | 2.0+ | Instantaneous rate for spike detection | Alerting rules (too spiky, flaps) | | `increase()` | 2.0+ | Total count increase over a window | Comparing across different window sizes | | `histogram_quantile()` | 2.0+ | Latency percentiles from histograms | Summary metrics (different type) | | `absent()` | 2.0+ | Alert when a metric stops being scraped | Checking if a value is zero (use `== 0`) | | `subquery` `[5m:1m]` | 2.3+ | Range query over instant vector function | Ad-hoc use — always create recording rule |

---

Correct Patterns

rate() Window Sizing

Use a window at least 4x the scrape interval. For a 15s scrape interval, minimum window is `1m`.

# Correct — 5m window with 15s scrape interval
rate(http_requests_total[5m])

# Correct — 1m minimum window for 15s scrape
rate(http_requests_total[1m])

**Why**: `rate()` requires at least 2 samples in the window to compute a slope. With a 15s scrape interval and a 15s window, you often get only 1 sample, returning no data or stale results.

---

histogram_quantile() with le Label

Always aggregate over the `le` label when using `histogram_quantile()`:

# Correct — aggregate over le bucket boundaries
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Correct — single service with explicit le
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket{service="api"}[5m])
)

**Why**: Omitting `le` in the aggregation collapses all buckets together, producing meaningless quantile values. The `le` label defines the bucket boundaries that `histogram_quantile()` uses for interpolation.

---

Recording Rules for Alert Expressions

Pre-compute expensive aggregations as recording rules named by convention `level:metric:ops`:

# recording_rules.yml
groups:
  - name: slo_rules
    interval: 30s
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job, status)

      - record: job:http_errors_total:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)

      - record: job:error_rate:ratio5m
        expr: |
          job:http_errors_total:rate5m
          /
          job:http_requests_total:rate5m

**Why**: Alert expressions evaluated every 15-30s against raw counters scan all samples in the window repeatedly. Recording rules pre-aggregate, reducing evaluation from O(N×samples) to O(1).

---

Pattern Catalog

<!-- no-pair-required: section header only -->

Use rate() Instead of irate() in Alert Rules

**Detection**:

grep -rn 'irate(' --include="*.yml" --include="*.yaml"
rg 'irate\(' --type yaml

<!-- no-pair-required: partial section — positive counterpart follows in next block -->

**Signal**:

# alert_rules.yml
- alert: HighErrorRate
  expr: irate(http_requests_total{status=~"5.."}[5m]) > 0.01

**Why this matters**: `irate()` uses only the last two samples, making it extremely sensitive to single-scrape spikes. Alert rules evaluated every 30s will flap on transient spikes, generating spurious notifications. Production alert fatigue follows.

**Preferred action**:

- alert: HighErrorRate
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
  for: 5m  # also add a for: clause to require sustained violation

---

Add a `for:` Clause on Latency Alerts

**Detection**:

grep -B5 'latency\|duration\|p99\|p95\|quantile' --include="*.yml" --include="*.yaml" -rn | grep -v 'for:'
rg 'alert:.*[Ll]atency' --type yaml -A 10 | grep -v 'for:'

**Signal**:

- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  # No for: clause!

**Why this matters**: Without `for:`, a single evaluation above the threshold fires the alert immediately. Network hiccups, deployment restarts, or pod scheduling create transient spikes that produce immediate pages. The signal-to-noise ratio degrades fast.

**Preferred action**:

- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m  # must sustain for 5 minutes before firing
  labels:
    severity: warning

---

Use Pre-Computed Quantile Labels for Summary Metrics

**Detection**:

grep -rn 'histogram_quantile.*_summary\|histogram_quantile.*quantile=' --include="*.yml"
rg 'histogram_quantile' --type yaml -A 3 | grep 'quantile='

<!-- no-pair-required: partial section — positive counterpart follows in next block -->

**Signal**:

# Wrong — using histogram_quantile on a summary type metric
histogram_quantile(0.99, rate(rpc_duration_seconds{quantile="0.99"}[5m]))

**Why this matters**: Summary metrics expose pre-computed quantiles (via the `quantile` label) that cannot be re-aggregated. Passing them to `histogram_quantile()` produces nonsense — the function expects `le` bucket labels, not pre-computed quantile values. The query may not error but will silently return wrong numbers.

**Preferred action**: Use the pre-computed quantile label directly:

# Correct