/slo-optimize

name: slo-optimize
description: |
  Analyzes Grafana SLO timeline trends via gcx and produces data-backed advisory recommendations:
  objective tuning, alerting sensitivity review, label visibility, or window adjustments.
  Use when the user wants to analyze SLO performance trends and receive improvement suggestions.
  Trigger on phrases like "optimize my SLO", "SLO improvement suggestions", "tune my SLO",
  "SLO performance analysis", or "should I change my SLO objective".
  For SLO status overview use slo-check-status.
  For investigating breaching SLOs use slo-investigate.
  For creating or modifying SLO definitions use slo-manage.
allowed-tools: Bash

SLO Optimizer

Analyze SLO timeline trends, compute statistics over the past 28 days, and generate advisory recommendations backed by real metric values. Never modify SLO definitions directly — route to slo-manage when the user wants to apply a recommendation.

Core Principles

1. Use gcx commands exclusively — do not call Grafana APIs directly. 2. Trust the user's expertise — skip explanations of what SLOs or burn rates are. 3. Use `-o json` for agent processing of structured output; default format for user display. 4. Show graph output for timeline data so the user can see the trend visually. 5. Every recommendation MUST include supporting data (current values, projected values, or historical comparisons). No generic advice without numbers. 6. This skill is advisory only. Route to slo-manage for any changes the user wants to apply.

Prerequisites

gcx configured with a context pointing to the target Grafana instance.

If the user does not supply a UUID, list available SLOs first:

gcx slo definitions list

Ask the user which SLO to analyze if the target is ambiguous.

Optimization Workflow

Step 1: Retrieve SLO Definition

gcx slo definitions get <UUID> -o json

Extract and note:

• `spec.name` — display name
• `spec.objectives[0].value` — current objective (e.g., 0.999)
• `spec.objectives[0].window` — compliance window (e.g., 28d)
• `spec.query.type` — ratio | freeform | threshold
• `spec.query.ratio.groupByLabels` — dimensional labels (may be empty)
• `spec.alerting` — fastBurn / slowBurn configuration (may be absent)
• `spec.destinationDatasource.uid` — datasource UID for metric queries

Step 2: Fetch 28-Day Timeline

# Default graph output for user display
gcx slo definitions timeline <UUID> --from now-28d --to now

# JSON output for statistical analysis
gcx slo definitions timeline <UUID> --from now-28d --to now -o json

Parse the JSON output to extract SLI values across the time series. Compute:

• `mean_sli` — average SLI over the 28-day window
• `min_sli` — lowest observed SLI point
• `max_sli` — highest observed SLI point
• `std_dev` — variability indicator

If timeline returns no data (NODATA), note it and skip to Step 3 for current status.

Step 3: Get Current Status (Wide Format)

gcx slo definitions status <UUID> -o wide

Extract from the wide output:

• Current SLI value
• Error budget remaining (%)
• Burn rate (current)
• SLI_1H and SLI_1D snapshots
• Status: OK | BREACHING | NODATA

Step 4: Query Raw SLI Metrics (When Timeline Is Insufficient)

When timeline data is sparse (< 7 days of points) or all NODATA, query raw metrics directly using the datasource UID from Step 1:

# SLI window metric (primary trend signal)
gcx metrics query -d <datasource-uid> \
  'grafana_slo_sli_window{grafana_slo_uuid="<UUID>"}' \
  --from now-28d --to now --step 6h

# Success and total rate for ratio SLOs
gcx metrics query -d <datasource-uid> \
  'grafana_slo_success_rate_5m{grafana_slo_uuid="<UUID>"}' \
  --from now-28d --to now --step 6h

gcx metrics query -d <datasource-uid> \
  'grafana_slo_total_rate_5m{grafana_slo_uuid="<UUID>"}' \
  --from now-28d --to now --step 6h

The recording rules label these series with `grafana_slo_uuid` (not `slo_uuid`). An empty result is not proof of a data gap — first rule out label, UUID, datasource, and time-range mismatches (these series live on the SLO's destination datasource, and a new SLO has no history yet). A bare `grafana_slo_sli_window` query discriminates quickly: series present means the selector is wrong; none at all points to a new SLO or the wrong datasource.

If the datasource UID is not in the definition, resolve it:

gcx datasources list --type prometheus

If the filtered list comes back empty, the stack may leave the `type` field blank in list payloads (known issue) — rerun without `--type` and pick the Prometheus datasource by name, or use the UID from `.spec.destinationDatasource.uid` directly.

Step 5: Analyze Trends

Classify the pattern using the timeline data from Steps 2 and 4:

**Sustained decline** — SLI trending downward for 7 or more consecutive days. Compute the slope over the last 7 days vs. the preceding 7 days to confirm direction.

• Recommendation trigger: investigate underlying service degradation; a window adjustment will

not fix a declining service.

**Periodic dips** — SLI drops recur at regular intervals (e.g., every weekend, every night). Look for temporal correlation in the min points.

• Recommendation trigger: window adjustment (e.g., 7d → 28d smooths weekend traffic spikes)

or objective reduction if the dips are expected.

**Sudden drops** — Step-change in SLI at a specific timestamp (deployment, config change). Identify the onset timestamp and estimate error budget consumed by the event.

• Recommendation trigger: check alerting is configured; if budget consumed > 20% by a single

event, consider tighter fastBurn thresholds.

**Budget exhaustion rate** — Project when the error budget will reach 0 based on the current burn rate from Step 3. Formula: `days_until_exhausted = budget_remaining_pct / (burn_rate * 100 / window_days)`

• Recommendation trigger: if < 7 days remain, flag as urgent; route to slo-investigate.

Step 6: Generate Advisory