/slo-investigate

name: slo-investigate
description: >-
  Diagnoses a breaching Grafana SLO via gcx: root cause, dimensional breakdown,
  alert rule correlation, runbooks. Use when an SLO is breaching or burning its
  error budget. For a status overview use slo-check-status. For tuning use
  slo-optimize.
allowed-tools: Bash

SLO Investigator

Deep-dive investigation of breaching SLOs: dimensional breakdown, alert correlation, runbook access. For experienced operators — no hand-holding.

Core Principles

1. Use gcx commands — do not call Grafana APIs directly (no curl, no HTTP libraries) 2. Trust the user's expertise — skip obvious context, get to the root cause 3. Use `-o json` for agent processing, default format for user display; show graphs for time-series data 4. Errors collected at the end — do not interleave error handling in workflow steps 5. Use `--from`/`--to` for all time-range commands (never `--start`/`--end`)

Investigation Workflow

Step 1: Retrieve SLO Definition

gcx slo definitions get <UUID> -o json

Extract from the JSON response:

• `.metadata.name` — SLO name
• `.spec.query.type` — query type: `ratio`, `freeform`, or `threshold`
• For ratio: `.spec.query.ratio.successMetric`, `.spec.query.ratio.totalMetric`, `.spec.query.ratio.groupByLabels[]`
• For freeform: `.spec.query.freeform.query`
• `.spec.objectives[0].value` — objective (0–1), `.spec.objectives[0].window` — window
• `.spec.destinationDatasource.uid` — Prometheus datasource UID
• `.spec.alerting.fastBurn.annotations`, `.spec.alerting.slowBurn.annotations` — runbook/dashboard URLs
• `.metadata.annotations` — additional runbook/dashboard references

If no UUID is given, list SLOs and ask which to investigate:

gcx slo definitions list

Step 2: Check Status with Wide Output

gcx slo definitions status <UUID> -o wide

This shows SLI, ERROR_BUDGET, BURN_RATE, SLI_1H, SLI_1D, and STATUS.

**Early exit — OK status:** If STATUS is OK, report health metrics and stop:

SLO: <name> — Status: OK
SLI: <value>  |  Error budget remaining: <budget>%  |  Burn rate: <rate>x
1h SLI: <sli_1h>  |  1d SLI: <sli_1d>
No action needed.

**Early exit — NODATA status:** If STATUS is NODATA, branch to NODATA diagnosis:

SLO: <name> — Status: NODATA
Recording rule metrics unavailable. Likely causes:
- Destination datasource misconfigured (check .spec.destinationDatasource.uid)
- Grafana recording rules not yet evaluated (can take 1–2 minutes after creation)
- Prometheus federation/remote write issue

Check: gcx datasources list --type prometheus
Then verify the destination datasource UID matches what the SLO expects.

When diagnosing NODATA, two environment realities save wasted queries: on remote-write stacks there are no scrape targets, so `up`-based scrape-health checks return nothing — absence of `up` series is not evidence of a problem. And if `datasources list --type prometheus` returns an empty list, the stack may leave `type` blank in list payloads (known issue) — rerun without `--type` or use the UID from `.spec.destinationDatasource.uid` directly.

**Lifecycle states:** If status is Creating/Updating/Deleting/Error, report that the SLO is in a transient state and investigate the Grafana backend.

Step 3: Render Timeline

gcx slo definitions timeline <UUID> --from now-1h --to now

For wider trends:

gcx slo definitions timeline <UUID> --from now-24h --to now

Show the graph output (default). Use it to identify when breaching started and how severe it is.

Step 4: Dimensional Breakdown

Resolve the datasource UID. If `.spec.destinationDatasource.uid` is set, use it. Otherwise auto-discover:

gcx datasources list --type prometheus

**For ratio queries** — extract success/total metric selectors and groupByLabels, then query dimensional breakdown:

# Success rate by dimension (e.g., cluster, status_code, endpoint)
gcx metrics query -d <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<successMetric>[5m])) / sum by (<groupByLabel>) (rate(<totalMetric>[5m]))' \
  --from now-1h --to now --step 1m

# Error rate by dimension to spot the bad actor
gcx metrics query -d <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<totalMetric>[5m])) - sum by (<groupByLabel>) (rate(<successMetric>[5m]))' \
  --from now-1h --to now --step 1m

If `groupByLabels` is empty, try common dimensions: `cluster`, `namespace`, `service`, `status_code`, `endpoint`.

**For freeform queries** — use the raw PromQL expression and add `by (<label>)` grouping:

gcx metrics query -d <datasource-uid> \
  '<freeform_expression> by (cluster)' \
  --from now-1h --to now --step 1m

# Also try other likely breakdown dimensions
gcx metrics query -d <datasource-uid> \
  '<freeform_expression> by (namespace)' \
  --from now-1h --to now --step 1m

Use graph output to display dimensional trends visually. Use `-o json` to extract exact values for the report.

To query the SLO's own recording rule metrics instead (`grafana_slo_sli_window`, burn rate, error budget expressions), see [references/slo-promql-patterns.md](references/slo-promql-patterns.md) for the metric inventory and ready-made PromQL patterns.

Step 5: Search for Related Alert Rules

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.name | test("<slo-name>"; "i"))]'

Also try searching by UUID fragment if the name-based search returns no results:

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.labels.grafana_slo_uuid == "<UUID>" or (.name | test("<slo-name>"; "i")))]'

Extract for each matching rule: name, state (firing/pending/inactive), labels, and annotations.

Step 6: Extract Runbook and Dashboard URLs

Collect URLs from:

• `.spec.alerting.fastBurn.annotations.runbook_url`
• `.spec.alerting.fastBurn.annotations.dashboard_url`
• `.spec.alerting.slowBurn.annotations.runbook_url`
• `.spec.alerting.slowBurn.annotations.dashboard_url`
• `.met