cardinality-management

Cardinality Management Reference

> **Scope**: Label cardinality detection, TSDB analysis, and relabeling to prevent OOM > **Version range**: Prometheus 2.0+ (TSDB analysis tools: 2.23+) > **Generated**: 2026-04-09 — cardinality budgets depend on Prometheus memory allocation

---

Overview

High cardinality is the #1 cause of Prometheus OOM in production. Each unique combination of label values creates one time series. A metric with `{service, endpoint, status, user_id}` and 1,000 users × 100 endpoints × 5 statuses = 500,000 series — from one metric. The failure mode is gradual: query performance degrades first, then OOM kills under query load. Detection must be proactive, not reactive.

---

Cardinality Budget Reference

| Memory Available | Safe Series Count | Warning Threshold | Critical Threshold | |------------------|-------------------|-------------------|--------------------| | 4 GB | ~1M series | 800K | 1.2M | | 8 GB | ~2M series | 1.6M | 2.4M | | 16 GB | ~4M series | 3.2M | 4.8M | | 32 GB | ~8M series | 6.4M | 9.6M |

Rule of thumb: Prometheus uses ~4KB per active time series for index + chunks in memory.

---

Detection Commands

Immediate Cardinality Check

# Total active series count (requires HTTP API access)
curl -s 'http://localhost:9090/api/v1/query?query=prometheus_tsdb_head_series' | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(d['data']['result'][0]['value'][1])"

# Top 20 metrics by series count
curl -s 'http://localhost:9090/api/v1/label/__name__/values' | \
  python3 -c "
import json, sys, subprocess, re
data = json.load(sys.stdin)
counts = []
for m in data['data'][:50]:  # sample first 50
    r = subprocess.run(['curl','-s', f'http://localhost:9090/api/v1/query?query=count({{__name__=\"{m}\"}} )'],
                       capture_output=True, text=True)
    val = json.loads(r.stdout)
    if val.get('data', {}).get('result'):
        counts.append((m, int(val['data']['result'][0]['value'][1])))
counts.sort(key=lambda x: -x[1])
for m, c in counts[:20]: print(f'{c:>10}  {m}')
"

# Prometheus TSDB analysis (requires promtool, Prometheus 2.23+)
promtool tsdb analyze /path/to/prometheus-data

# Check series count by metric name via PromQL
topk(20, count by (__name__)({__name__=~".+"}))

---

Pattern Catalog

<!-- no-pair-required: section header only -->

Use Bounded Labels Only

**Detection**:

# Find instrumentation code with potentially unbounded label values
grep -rn 'user_id\|request_id\|session_id\|trace_id\|transaction_id' \
  --include="*.go" --include="*.py" --include="*.js" | grep -i 'label\|metric\|prometheus'

rg 'WithLabelValues|labels\.Set|prometheus\.Labels' --type go -A 3 | \
  grep -i 'user\|request_id\|session\|trace'

# Check actual cardinality of suspect metrics in PromQL
count by (user_id) (http_requests_total)  # should return 0 results if correctly designed

**Signal**:

// Go example — unbounded label
httpRequests.With(prometheus.Labels{
    "user_id": userID,      // WRONG: unbounded
    "endpoint": endpoint,
    "status":  strconv.Itoa(statusCode),
}).Inc()

**Why this matters**: 100K users × 50 endpoints × 5 status codes = 25M series. Prometheus memory usage becomes `25M × 4KB = 100GB`. Queries against this metric scan all 25M series even with filters, because index lookup is by label, not by value range.

**Preferred action**:

// Correct — bounded labels only
httpRequests.With(prometheus.Labels{
    "endpoint": endpoint,   // bounded: known set of routes
    "status":  statusCode,  // bounded: 2xx/3xx/4xx/5xx
    "method":  r.Method,    // bounded: GET/POST/PUT/DELETE
}).Inc()
// Track per-user analytics in a separate system (Kafka, ClickHouse)

---

Add Relabeling Drop Rules for Internal Metrics

**Detection**:

# Check if prometheus.yml has any drop relabeling rules
grep -n 'action: drop\|action: keep' prometheus.yml
# If no results: no cardinality guardrails in scrape config

# Check what labels are coming in from a target
curl -s 'http://localhost:9090/api/v1/targets' | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [print(t['labels']) for t in d['data']['activeTargets'][:5]]"

<!-- no-pair-required: partial section — positive counterpart follows in next block -->

**Signal**:

# prometheus.yml — no relabeling
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    # No relabel_configs — ingests all labels from pod annotations

**Why this matters**: Kubernetes pods can expose dozens of labels (app, version, helm-release, git-commit, build-time, namespace). Without `relabel_configs`, all of these become Prometheus label dimensions. A git commit hash label creates unique series per deployment, exploding cardinality.

**Preferred action**:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only the labels you actually need
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      # Drop all other __meta_ labels (they're huge and mostly unused)
      - regex: __meta_kubernetes_.*
        action: labeldrop
      # Drop pods with no app label (system pods you don't care about)
      - source_labels: [app]
        regex: .+
        action: keep

---

Exclude High-Cardinality Labels from Recording Rules

**Detection**:

# Find recording rules that preserve high-cardinality labels
grep -A 5 'record:' prometheus-rules.yml | grep 'by (' | grep -i 'user\|request_id\|pod_name'
rg 'record:' --type yaml -A 5 | grep 'by\s*\(' | grep -v 'service\|job\|namespace\|status'

**Signal**:

- record: job:http_requests:rate5m
  expr: sum(rate(http_requests_total[5m])) by (job, user_id)
  # Aggregates but still fans out by user_id — doesn't help

**Why this matters**: Recording rules ar