/prometheus-label-strategy

name: prometheus-label-strategy
license: Apache-2.0
description: >
  Expert evaluator for Prometheus label strategy on Grafana Cloud. Audits, designs, and
  improves label schemas using cardinality scoring, access-pattern alignment, static vs.
  dynamic label rules, histogram bucket discipline, and instrumentation hygiene. Prevents
  high cardinality at the source — in application code and scrape target labels — without
  dropping labels that make series unique (which breaks the data). For reducing the cost of
  series already in Grafana Cloud, routes to the adaptive-metrics skill. Use when the user
  asks to evaluate, audit, design, or improve Prometheus labels — or asks how to prevent
  high cardinality at the source. For "why is my Prometheus slow / expensive right now"
  triage, see prometheus-cardinality-troubleshooter.

Prometheus Label Strategy Evaluator

You are an expert in Prometheus label strategy. When asked to evaluate, audit, design, or improve a Prometheus label schema — or when a user asks how to prevent high cardinality at the source — use this guide to provide structured, actionable advice.

This skill is about **preventing bad labels at the source** — in application instrumentation and in scrape *target* labels — so they never enter storage. It is **not** about stripping labels off metrics after they've been emitted: removing a label that makes a series unique at scrape time silently breaks the data (see [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) below). For reducing the cost of series that already exist in Grafana Cloud, route the user to the `adaptive-metrics` skill. For diagnosing an active cardinality fire, route to `prometheus-cardinality-troubleshooter`.

---

The One Rule: Never Drop a Label That Makes a Series Unique

**You cannot remove, at scrape time, any label that makes a series unique.** Not `pod`, not `instance`, not anything that distinguishes one real series from another. This includes `metric_relabel_configs` with `action: labeldrop` and the equivalent `prometheus.relabel` rules in Alloy.

It looks like a cardinality win. It is not — it **breaks the data**, silently and permanently:

• **Counter resets get mixed together.** When two pods' counters collapse into one series, their independent restarts interleave on the merged series. `rate()` and `increase()` then return garbage — often *absurdly high* values, because every pod restart looks like a counter reset.
• **DPM inflates instead of dropping.** Multiple samples now land on the same series in the same scrape — duplicate samples, out-of-order errors, inflated samples-per-minute. People come back weeks later asking "why is my DPM so high?" or "why is `rate()` returning absurd numbers?" — and there is **no evidence left in the data** of where it broke.
• **The aggregation is wrong, not just coarse.** A `sum` over a label you dropped silently double-counts or under-counts depending on how the collapse happened.

The trap is that none of this errors at config time. The pipeline keeps running; the numbers are just quietly wrong, and the breakage point is invisible after the fact.

**The right tools, in order:**

1. **Don't emit the bad label in the first place** — fix the application code. This is the only place a label can be *removed* without consequence, because the series was never unique on it to begin with. 2. **For series already flowing into Grafana Cloud that you can't fix at the source → Adaptive Metrics.** This is exactly what it is for: it aggregates series *correctly* — counter-reset-aware, with a recorded audit trail, and reversible — instead of blindly stripping labels. Route the user to the `adaptive-metrics` skill.

`metric_relabel_configs` has a couple of narrow, safe uses (dropping an *entire* unwanted metric; removing a label that *exactly duplicates* a target label) — covered in [Source-Side Prevention](#4-metric_relabel_configs-narrow-safe-uses-only) — but **reducing cardinality by dropping a distinguishing label is never one of them.**

---

Core Concepts

**Series** are the fundamental unit in Prometheus. Each unique combination of metric name plus label key-value pairs creates a new active series. Too many series = memory pressure, slow queries, ingest pressure, high bill.

**Cardinality** = the number of unique values a label can have. Total series for a metric ≈ the *product* of cardinalities across its labels. A metric with `path` (100 values), `status_code` (10 values), `method` (5 values), and `instance` (50 values) = **250,000 series per metric**. Adding one more high-cardinality label often 10–100×s the count.

**The dual impact rule**: High-cardinality labels hurt on both paths:

• **Ingestion path**: More active series → larger head block, larger WAL, more memory, larger remote_write payloads, higher Grafana Cloud bill (Active Series + DPM)
• **Query path**: PromQL operators (`sum by`, `rate`, joins) must materialize matching series in memory. High cardinality balloons query memory and latency

**Series churn** is the silent killer. If a label value changes frequently (deploy version, pod name, ephemeral IDs), every change creates a *new* series while the old one continues to age out. Daily churn of 100% means you carry roughly 2× the steady-state series count for retention purposes.

**The key question for any proposed label**: "Will queries that use this metric reliably specify or aggregate on this label?" If no → it should NOT be a label.

---

Label Evaluation Framework

When auditing a label set, assess each label against these criteria.

Cardinality Scoring

| Label Example | Cardinality | Verdict | |---|---|---| | `env` (prod/staging/dev) | 2–5 values | ✅ Good | | `job` (Prometheus scrape job) | 5–50 values | ✅ Good | | `cluster`, `region` | Tens | ✅ Good | | `namespace` (K8s) | Tens–low hundreds | ✅ Acceptable | | `service`, `workload`, `container` | Tens–hundreds | ✅ Acceptable | | `instance` (host:port) |