/devops-observability

name: devops-observability
description: Use when designing, writing, reviewing or debugging monitoring, logging or tracing - Prometheus (scrape config, service discovery, relabeling, recording and alerting rules, retention, remote write, cardinality), PromQL and histograms, Alertmanager routing/grouping/inhibition/silences, exporters (node, blackbox, cAdvisor, kube-state-metrics), kube-prometheus-stack and Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule), long-term storage choices (Thanos, Mimir, VictoriaMetrics), Grafana provisioning as code, Loki with Grafana Alloy (Promtail is end of life), LogQL, OpenTelemetry Collector pipelines, sampling, Tempo/Jaeger, alert design and observability troubleshooting. Not for SLO and error-budget method or incident response (devops-sre-ops), Kubernetes workloads (devops-kubernetes), host OS tuning (devops-linux) or SaaS vendor products.
versions:
  prometheus: "3.16.0 (2026-11-04); LTS 3.13 (supported to 2027-07-31)"
  alertmanager: "0.35.0 (2026-11-09)"
  grafana: "13.3.0"
  loki: "3.7.9"
  grafana-alloy: "1.21.3 (2026-10-28)"
  opentelemetry-collector: "0.165.0 (core 1.71.0)"
  prometheus-operator: "0.95.1 (2026-11-05); kube-prometheus-stack chart 93.1.1 (main)"
  exporters: "node_exporter 1.12.1, blackbox_exporter 0.29.1, cAdvisor 0.60.6, kube-state-metrics 2.20.0"
user-invocable: true
references:
  - references/signals-and-method.md
  - references/prometheus-config.md
  - references/promql-essentials.md
  - references/alertmanager.md
  - references/exporters.md
  - references/prometheus-operator-k8s.md
  - references/long-term-storage.md
  - references/grafana-as-code.md
  - references/logs-loki-alloy.md
  - references/logs-other-stacks.md
  - references/tracing-opentelemetry.md
  - references/alert-design.md
  - references/observability-safe-ops.md
  - references/troubleshooting-playbook.md
  - references/templates/prometheus.yml
  - references/templates/file-sd-nodes.yml
  - references/templates/blackbox.yml
  - references/templates/rules-alerts.yml
  - references/templates/rules-recording.yml
  - references/templates/alertmanager.yml
  - references/templates/servicemonitor.yaml
  - references/templates/prometheusrule.yaml
  - references/templates/kube-prometheus-stack-values.yaml
  - references/templates/grafana-datasources.yaml
  - references/templates/grafana-dashboards-provider.yaml
  - references/templates/grafana-alerting-routing.yaml
  - references/templates/loki-config.yaml
  - references/templates/alloy-host.alloy
  - references/templates/alloy-k8s-logs.alloy
  - references/templates/alloy-otlp.alloy
  - references/templates/otel-collector.yaml
  - references/templates/docker-compose-monitoring.yaml
related-skills:
  - devops-sre-ops
  - devops-kubernetes
  - devops-k8s-cluster
  - devops-gitops-packaging
  - devops-docker
  - devops-linux
  - devops-networking
  - devops-storage-backup
  - devops-virtualization

<objective> Covers the three observability signals for an IT/DevOps estate: Prometheus metrics and rules, Alertmanager routing, exporters, the Kubernetes Operator stack, Grafana as code, Loki logs collected by Grafana Alloy, OpenTelemetry tracing and sampling, alert design, and troubleshooting the pipeline itself. Does not cover SLO/error-budget design and incident response (devops-sre-ops), workload manifests (devops-kubernetes) or SaaS vendor products. </objective>

DevOps Observability

Agent Workflow (MANDATORY)

Before ANY monitoring config or change, spawn in parallel, one `Agent` call each with a `name`:

1. **fuse-ai-pilot:explore-codebase** - find existing `prometheus.yml`, rule files, Alertmanager/Loki/Alloy/Collector configs, Helm values, dashboards, ServiceMonitors 2. **fuse-ai-pilot:research-expert** - verify option names for the deployed version at prometheus.io, grafana.com/docs, opentelemetry.io (the OpenTelemetry Collector renames components between 0.x releases) 3. **mcp__context7__query-docs** - Prometheus, Grafana, Loki, OpenTelemetry docs

Then discover read-only (running versions, `promtool check`, `amtool check-config`, `alloy validate`). After writing files, run **fuse-ai-pilot:sniper**. Live changes follow the agent's Operations Safety Protocol and [observability-safe-ops.md](references/observability-safe-ops.md).

Versions (checked 2026-11-10)

Prometheus 3.16.0 is latest; **LTS 3.13** is the conservative choice. Alertmanager 0.35.0 has breaking flag/config changes (event recorder outputs need a `name`). **Promtail reached end of life on 2026-03-02**: collect logs with Grafana Alloy. The OpenTelemetry Collector renamed `otlp` to `otlp_grpc`, `otlphttp` to `otlp_http`, and added a `queue_batch` processor meant to replace `batch`; read its changelog before every 0.x bump.

Overview

| Need | Decision in one line | |------|----------------------| | **What to instrument** | USE for resources, RED for services, golden signals at the edge; plus a blackbox probe of every user-facing URL | | **Metrics store** | Prometheus (local TSDB, retention by time AND size flags); long-term via remote write to Mimir/Thanos/VictoriaMetrics only when needed | | **Alerts** | Symptom-based, `for:` + `keep_firing_for`, severity `page`/`ticket`, every alert has `runbook_url`; routing and inhibition in Alertmanager | | **Kubernetes** | kube-prometheus-stack; ServiceMonitor/PodMonitor/PrometheusRule; check the `release` label selectors and PVCs | | **Logs** | Loki + Alloy; few static labels, rest in structured metadata; retention by the compactor | | **Traces** | OpenTelemetry SDK -> Collector/Alloy (OTLP 4317/4318) -> Tempo; tail-sample errors and slow traces | | **Dashboards/alerts as code** | Grafana provisioning files in Git; metric alerts live in Prometheus, not Grafana |

Critical Rules

1. **Version first** - read the running versions and the config of the target before writing an option; renames and removals are frequent (OpenTelemetry Collector, Alertmanager, Prometheus 3.x) 2.