/go-observability

name: go-observability
description: "Use when instrumenting Go services with metrics and distributed traces, or wiring exemplars and request-id propagation. Covers Prometheus patterns (Counter/Gauge/Histogram, low-cardinality labels), OpenTelemetry tracing (TracerProvider, span attributes, errors, context propagation), and metric ↔ trace correlation so a P99 spike jumps to the offending trace. Logging: see go-logging."
license: MIT
compatibility: "Designed for Claude Code or similar AI coding agents. Requires Go 1.21+ (for log/slog context variants used in correlation snippets). Prometheus client_golang and OpenTelemetry Go SDK."
allowed-tools: Read Edit Write Glob Grep Bash(go:*) Bash(golangci-lint:*)

Go Observability — Metrics, Traces, and Correlation

Production Go services need at least two always-on signals to be debuggable: **metrics** (aggregated measurements for alerting and SLOs) and **traces** (per-request flow showing where time went). The third deliverable is **correlation**: a P99 metric spike must lead, in one click, to the trace that caused it.

> **Logging is not in this skill.** Structured logging with `log/slog`, log levels, and zap/logrus/zerolog migration belong to the **go-logging** skill. This skill only references logs in the context of correlating them with traces.

Core Rules

1. **A feature is not done until it is observable.** New code MUST export at least: one Counter for operations, one Counter for errors, one Histogram for latency. 2. **Histograms, not Summaries, for latency.** Summaries cannot be aggregated across instances; Histograms support `histogram_quantile()` server-side. 3. **Label cardinality is bounded.** Never put unbounded values (user IDs, full URLs, request IDs) in Prometheus labels. Use route patterns, status classes, method. 4. **Context flows everywhere.** A function that does I/O takes `ctx context.Context` as its first argument. No context = no trace propagation = no correlation. 5. **Record errors on the span.** When a span ends in failure, call `span.RecordError(err)` and `span.SetStatus(codes.Error, ...)`. A green span hides a real failure. 6. **Correlate or it didn't happen.** Inject `trace_id` into logs, attach exemplars to histograms. Otherwise the three signals are three different products.

Signal Decision

Pick the signal that matches the question. Do not log what should be a metric.

| Question | Signal | Tool | |---|---|---| | How often does X happen? Error rate? Rate-per-second? | Metric (Counter) | Prometheus | | What is the P99 latency of endpoint /orders? | Metric (Histogram) | Prometheus + `histogram_quantile` | | Where did this one slow request spend its time? | Trace | OpenTelemetry | | Why does latency spike at 14:32? | Metric → exemplar → trace | Prometheus + OTel + exemplars | | What concrete error message did this request hit? | Log (see go-logging) | `log/slog` |

> Read [references/metrics.md](references/metrics.md) for Counter/Gauge/Histogram patterns, naming, and PromQL-as-comments. > Read [references/tracing.md](references/tracing.md) for TracerProvider setup, span attributes, and `otelhttp` middleware.

Metrics — the 60-Second Setup

import "github.com/prometheus/client_golang/prometheus"

// rate(http_requests_total{code=~"5.."}[5m]) / rate(http_requests_total[5m])
var httpRequests = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests by method, route, status class.",
    },
    []string{"method", "route", "code"}, // ALL bounded
)

// histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
var httpLatency = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "HTTP request latency in seconds.",
        Buckets: prometheus.DefBuckets,
    },
    []string{"method", "route"},
)

The comment above each metric is the PromQL it is designed to answer. This makes the metric discoverable and grep-able from a dashboard.

Traces — the 60-Second Setup

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/codes"
)

func (s *OrderService) Create(ctx context.Context, in CreateOrderInput) (*Order, error) {
    ctx, span := otel.Tracer("order-service").Start(ctx, "OrderService.Create")
    defer span.End()
    span.SetAttributes(attribute.String("order.user_id", in.UserID))

    order, err := s.repo.Insert(ctx, in)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "insert failed")
        return nil, fmt.Errorf("creating order: %w", err)
    }
    return order, nil
}

Every service method, every DB query, every external HTTP call gets a span. Context **must** flow into `s.repo.Insert(ctx, ...)` so the DB span is a child of the service span.

Correlation

Metrics → Traces with Exemplars

An exemplar attaches a single trace_id to a histogram observation. In Grafana, click the dot on a P99 spike and you land on the trace.

obs := httpLatency.WithLabelValues(r.Method, routePattern)
sc := trace.SpanContextFromContext(ctx)
if eo, ok := obs.(prometheus.ExemplarObserver); ok && sc.IsValid() {
    eo.ObserveWithExemplar(elapsed.Seconds(),
        prometheus.Labels{"trace_id": sc.TraceID().String()})
} else {
    obs.Observe(elapsed.Seconds())
}

Logs → Traces

Use the `otelslog` bridge (see go-logging skill) so every `slog.InfoContext(ctx, ...)` call automatically emits `trace_id` and `span_id`. You can then grep logs by trace_id when starting from a trace, or jump from a log line to the trace.

> Read [references/correlation.md](references/correlation.md) for exemplar wiring details, request-id propagation, and the end-to-end "metric spike → trace → log" workflow.

Context Propagation

// Bad — breaks trace propagation; the DB call starts a new root trace.
result, err := db.Query("SELECT ...")

// Good — the DB span is a child of