/debug-with-grafana

name: debug-with-grafana
description: >
  Structured workflow for investigating application problems with Grafana
  observability data (metrics, logs, traces) via gcx. Covers live
  firefighting AND retrospective incident analysis: incident triage,
  root-cause analysis, blast-radius checks (did an incident spill into
  other services), verifying whether a deployment or rollout triggered an
  incident, finding which service, endpoint, or path owns the most errors
  or slow requests, checking whether retries or queue backlogs piled up,
  and quantifying error or latency shares over a time window. Trigger on:
  "my API is returning 500 errors", "latency is spiking", "investigate why
  requests are failing", "triage the incident", "blast radius", "root
  cause", "did the rollout cause it", "which endpoint owns the most 5xx",
  "did retries pile up", or any request to analyse an earlier incident
  window using telemetry. For authoring dashboards use create-dashboard;
  for dashboard inventory use manage-dashboards.

Debug with Grafana

A structured 7-step diagnostic workflow for debugging application issues using Prometheus metrics, Loki logs, and Grafana resources. Follow steps in order — each step informs the next.

Prerequisites

gcx must be installed and configured with a valid context before running any commands. If not configured, use the `setup-gcx` skill first:

# Verify configuration
gcx config view

# Switch context if needed
gcx config use-context <context-name>

Diagnostic Workflow

Step 1: Discover Datasources

List all available datasources to identify Prometheus and Loki UIDs. All subsequent query commands require a datasource UID via `-d <uid>`.

# List all datasources
gcx datasources list -o json

# Filter by type for scripting
gcx datasources list -t prometheus -o json
gcx datasources list -t loki -o json

# Capture UIDs for use in subsequent steps
PROM_UID=$(gcx datasources list -t prometheus -o json 2>/dev/null | \
  python3 -c "import json,sys; print(json.load(sys.stdin)['datasources'][0]['uid'])")
LOKI_UID=$(gcx datasources list -t loki -o json 2>/dev/null | \
  python3 -c "import json,sys; print(json.load(sys.stdin)['datasources'][0]['uid'])")

**Expected output shape:**

{
  "datasources": [
    {"uid": "<uid>", "name": "<display-name>", "type": "prometheus", ...},
    {"uid": "<uid>", "name": "<display-name>", "type": "loki", ...}
  ]
}

If no datasources appear, confirm the context is pointing at the correct Grafana instance. See `references/error-recovery.md` for auth and datasource-not-found recovery patterns.

> **JSON output piping**: When piping gcx output through external tools, never > use `2>&1` — gcx writes hints to stderr that break JSON parsers. Use > `2>/dev/null` to suppress stderr, or use `--json field1,field2` to select > fields directly without piping: > ```bash > gcx datasources list -t prometheus --json uid > gcx metrics query -d <prom-uid> 'up' --json metric,value > ``` > Use `--json list` to discover available fields for any command.

Step 2: Confirm Data Availability

Before querying specific metrics, confirm the target service is instrumented and data is flowing. This avoids wasting time on empty results.

# Check that the target service is being scraped
gcx metrics query -d <prom-uid> 'up' -o json

# Verify the relevant job label exists
gcx metrics labels -d <prom-uid> -l job -o json

# For Loki: confirm log streams exist for the service
gcx logs labels -d <loki-uid> -l job -o json
gcx logs series -d <loki-uid> -M '{job="<service-name>"}' -o json

# Spot-check: confirm uptime metrics are present for the service
gcx metrics query -d <prom-uid> 'up{job="<service-name>"}' -o json

**Expected output shape:**

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {"metric": {"__name__": "up", "job": "<service-name>", "instance": "<host:port>"}, "value": [<timestamp>, "<0-or-1>"]}
    ]
  }
}

A `value` of `"0"` means the service is down or not being scraped. Empty `result` array means the metric is absent — see Failure Mode 3 in `references/error-recovery.md`.

Step 3: Query Error Rates

Query the HTTP 5xx error rate over the relevant time window to establish whether an error spike exists and when it began.

# HTTP 5xx error rate (range query for trend)
gcx metrics query -d <prom-uid> \
  'rate(http_requests_total{job="<service-name>",status=~"5.."}[5m])' \
  --from now-1h --to now --step 1m -o json

# Visualize the trend
gcx metrics query -d <prom-uid> \
  'rate(http_requests_total{job="<service-name>",status=~"5.."}[5m])' \
  --from now-1h --to now --step 1m -o graph

# Error ratio (errors / total)
gcx metrics query -d <prom-uid> \
  'rate(http_requests_total{job="<service-name>",status=~"5.."}[5m]) / rate(http_requests_total{job="<service-name>"}[5m])' \
  --from now-1h --to now --step 1m -o json

# Break down by status code to identify 500 vs 503 vs 504
gcx metrics query -d <prom-uid> \
  'sum by(status) (rate(http_requests_total{job="<service-name>"}[5m]))' \
  --from now-1h --to now --step 1m -o json

**Expected output shape (matrix for range queries):**

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {"job": "<service-name>", "status": "<code>"},
        "values": [[<timestamp>, "<rate>"], ...]
      }
    ]
  }
}

Note the timestamp where the rate increases — this is the incident start time. Use this window in subsequent steps.

Step 4: Query Latency

Query request latency to determine whether the service is slow (latency issue) or failing fast (error issue). High latency often precedes error spikes.

# P50/P95/P99 latency from histogram
gcx metrics query -d <prom-uid> \
  'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="<service-name>"}[5m]))' \
  --from now-1h --to now --step 1m -o json

# Visualize P95 l