/slo-investigate

name: slo-investigate
description: Diagnoses breaching Grafana SLOs via gcx - root cause analysis, dimensional breakdown, alert rule correlation, runbook access. Use when a specific SLO is breaching or alerting and the user needs to understand why. Trigger on phrases like "investigate SLO", "why is my SLO breaching", "SLO error budget burning", "SLO alerting". For SLO status overview use slo-check-status. For creating or modifying SLOs use slo-manage. For optimization suggestions use slo-optimize.
allowed-tools: Bash

SLO Investigator

Deep-dive investigation of breaching SLOs: dimensional breakdown, alert correlation, runbook access. For experienced operators — no hand-holding.

Core Principles

1. Use gcx commands — do not call Grafana APIs directly (no curl, no HTTP libraries) 2. Trust the user's expertise — skip obvious context, get to the root cause 3. Use `-o json` for agent processing, default format for user display; show graphs for time-series data 4. Errors collected at the end — do not interleave error handling in workflow steps 5. Use `--from`/`--to` for all time-range commands (never `--start`/`--end`)

Investigation Workflow

Step 1: Retrieve SLO Definition

gcx slo definitions get <UUID> -o json

Extract from the JSON response:

• `.metadata.name` — SLO name
• `.spec.query.type` — query type: `ratio`, `freeform`, or `threshold`
• For ratio: `.spec.query.ratio.successMetric`, `.spec.query.ratio.totalMetric`, `.spec.query.ratio.groupByLabels[]`
• For freeform: `.spec.query.freeform.query`
• `.spec.objectives[0].value` — objective (0–1), `.spec.objectives[0].window` — window
• `.spec.destinationDatasource.uid` — Prometheus datasource UID
• `.spec.alerting.fastBurn.annotations`, `.spec.alerting.slowBurn.annotations` — runbook/dashboard URLs
• `.metadata.annotations` — additional runbook/dashboard references

If no UUID is given, list SLOs and ask which to investigate:

gcx slo definitions list

Step 2: Check Status with Wide Output

gcx slo definitions status <UUID> -o wide

This shows SLI, ERROR_BUDGET, BURN_RATE, SLI_1H, SLI_1D, and STATUS.

**Early exit — OK status:** If STATUS is OK, report health metrics and stop:

SLO: <name> — Status: OK
SLI: <value>  |  Error budget remaining: <budget>%  |  Burn rate: <rate>x
1h SLI: <sli_1h>  |  1d SLI: <sli_1d>
No action needed.

**Early exit — NODATA status:** If STATUS is NODATA, branch to NODATA diagnosis:

SLO: <name> — Status: NODATA
Recording rule metrics unavailable. Likely causes:
- Destination datasource misconfigured (check .spec.destinationDatasource.uid)
- Grafana recording rules not yet evaluated (can take 1–2 minutes after creation)
- Prometheus federation/remote write issue

Check: gcx datasources list --type prometheus
Then verify the destination datasource UID matches what the SLO expects.

When diagnosing NODATA, two environment realities save wasted queries: on remote-write stacks there are no scrape targets, so `up`-based scrape-health checks return nothing — absence of `up` series is not evidence of a problem. And if `datasources list --type prometheus` returns an empty list, the stack may leave `type` blank in list payloads (known issue) — rerun without `--type` or use the UID from `.spec.destinationDatasource.uid` directly.

**Lifecycle states:** If status is Creating/Updating/Deleting/Error, report that the SLO is in a transient state and investigate the Grafana backend.

Step 3: Render Timeline

gcx slo definitions timeline <UUID> --from now-1h --to now

For wider trends:

gcx slo definitions timeline <UUID> --from now-24h --to now

Show the graph output (default). Use it to identify when breaching started and how severe it is.

Step 4: Dimensional Breakdown

Resolve the datasource UID. If `.spec.destinationDatasource.uid` is set, use it. Otherwise auto-discover:

gcx datasources list --type prometheus

**For ratio queries** — extract success/total metric selectors and groupByLabels, then query dimensional breakdown:

# Success rate by dimension (e.g., cluster, status_code, endpoint)
gcx metrics query -d <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<successMetric>[5m])) / sum by (<groupByLabel>) (rate(<totalMetric>[5m]))' \
  --from now-1h --to now --step 1m

# Error rate by dimension to spot the bad actor
gcx metrics query -d <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<totalMetric>[5m])) - sum by (<groupByLabel>) (rate(<successMetric>[5m]))' \
  --from now-1h --to now --step 1m

If `groupByLabels` is empty, try common dimensions: `cluster`, `namespace`, `service`, `status_code`, `endpoint`.

**For freeform queries** — use the raw PromQL expression and add `by (<label>)` grouping:

gcx metrics query -d <datasource-uid> \
  '<freeform_expression> by (cluster)' \
  --from now-1h --to now --step 1m

# Also try other likely breakdown dimensions
gcx metrics query -d <datasource-uid> \
  '<freeform_expression> by (namespace)' \
  --from now-1h --to now --step 1m

Use graph output to display dimensional trends visually. Use `-o json` to extract exact values for the report.

To query the SLO's own recording rule metrics instead (`grafana_slo_sli_window`, burn rate, error budget expressions), see [references/slo-promql-patterns.md](references/slo-promql-patterns.md) for the metric inventory and ready-made PromQL patterns.

Step 5: Search for Related Alert Rules

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.name | test("<slo-name>"; "i"))]'

Also try searching by UUID fragment if the name-based search returns no results:

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.labels.grafana_slo_uuid == "<UUID>" or (.name | test("<slo-name>"; "i")))]'

Extract for each matching rule: name, state (firing/pending/inactive), labels, and annotations.

Step 6: Extract Runbook and Dashboard URLs

Collect URLs from:

• `.s