/slo-manage

name: slo-manage
description: >-
  Creates, updates, pulls, pushes, and deletes Grafana SLO definitions via gcx,
  with dry-run validation and GitOps sync. For SLO health use slo-check-status.
  For a breaching SLO use slo-investigate.
allowed-tools: Bash, Read, Write, Edit

SLO Management

Create, update, sync, and delete SLO definitions using gcx.

Core Principles

1. Use gcx commands exclusively — do not call Grafana APIs directly 2. Always run `--dry-run` before any push operation; proceed only if dry-run succeeds 3. Trust the user's expertise — skip explanations of SLO concepts 4. Use `-o json` for agent processing; default table/yaml for user display 5. Auto-resolve datasource UIDs; only ask if auto-discovery fails

Query Type Decision Table

Select query type based on what the user describes:

| User describes | Query type | |----------------|------------| | "percentage of successful requests", "success rate", "error rate" | `ratio` | | "raw PromQL expression", "custom metric formula" | `freeform` | | "metric above/below threshold", "latency under X ms", "availability percentage" | `threshold` |

Metric-Pattern Decision Table

Use the metric name suffix to pick the query type when the user provides a metric name:

| Metric suffix / type | Query type | Rationale | |----------------------|------------|-----------| | `_total` counter | `ratio` | success_total / all_total | | `_bucket` histogram | `threshold` | use le-bound threshold on quantile | | `_gauge` or `up` metric | `threshold` | compare to fixed threshold | | None of the above | `freeform` | last resort only |

**Guardrail:** Freeform is a last resort. Before choosing freeform, verify the SLI cannot be expressed as ratio or threshold.

**Hard requirement:** Freeform queries MUST use `$__rate_interval` in all `rate()`/`increase()` calls. Literal ranges like `[5m]` are rejected by the SLO API.

Workflow 1: Create New SLO

Step 1: Determine query type using the decision table above

Step 2: Resolve destination datasource UID

gcx datasources list --type prometheus

Use the UID from the output. Some stacks do not populate `type` in the list response, so the filter can come back empty even though Prometheus datasources exist — in that case list everything and pick the canonical Grafana Cloud Prometheus entry (name like `grafanacloud-<stack>-prom`):

gcx datasources list

If that leaves zero or multiple plausible Prometheus candidates, ask the user which UID to use rather than guessing (the stack's `default: true` datasource is not necessarily Prometheus).

Step 3: Build YAML from the appropriate template

For complete ratio, freeform, and threshold templates (including alerting, labels, and folder fields), see [references/slo-templates.md](references/slo-templates.md). Key structure:

apiVersion: slo.ext.grafana.app/v1alpha1
kind: SLO
metadata:
  name: ""        # leave empty for new SLO (server assigns UUID on create)
spec:
  name: "my-api-availability"
  description: "API availability over 28 days"
  query:
    type: ratio   # freeform | ratio | threshold
    ratio:        # field matches type
      successMetric:
        prometheusMetric: http_requests_total{status!~"5.."}
      totalMetric:
        prometheusMetric: http_requests_total
      groupByLabels: [cluster, service]
  objectives:
    - value: 0.999   # 0.9 to 0.9999 typical range
      window: 28d    # 7d | 14d | 28d | 30d
  destinationDatasource:
    uid: <prometheus-uid>

Step 4: Preview with dry-run, then push

gcx resources push slos -p slo.yaml --dry-run
gcx resources push slos -p slo.yaml

**Push semantics:**

• `metadata.name` set to UUID → update if it exists; otherwise match by the SLO name before creating.
• Empty `metadata.name` → match by the SLO name before creating; the server assigns a UUID only when creating.
• A dry-run against an adapter without server validation is reported as unverified and sends no writes; it does not establish that the backend accepts the manifest.

Run `gcx slo definitions list` to confirm.

Workflow 2: Update Existing SLO

Step 1: Get current definition

gcx slo definitions get <UUID> -o yaml > slo.yaml

Step 2: Modify the YAML file

Edit the relevant fields (objective value, query, alerting, etc.). Do not modify `metadata.name` (UUID) or `readOnly` fields.

Step 3: Dry-run, then push

gcx resources push slos -p slo.yaml --dry-run
gcx resources push slos -p slo.yaml

Workflow 3: GitOps Sync (Pull/Push)

Pull all SLOs to disk

gcx resources pull slos -p ./slos -o yaml
# Writes to ./slos/slos.v1alpha1.slo.ext.grafana.app/<uuid>.yaml

Push directory of SLOs

gcx resources push slos -p ./slos --dry-run
gcx resources push slos -p ./slos

Workflow 4: Delete SLO

Step 1: Confirm SLO identity

gcx slo definitions list
gcx slo definitions get <UUID>

Confirm the UUID and name with the user before deletion.

Step 2: Delete

gcx slo definitions delete <UUID> --force

Use `--force` to skip the confirmation prompt when running in agent mode (there is no `-f` shorthand on delete).

Configuration Guidance

**Objective values** (stored as 0–1, displayed as percentage):

• Typical range: 0.9 (90%) to 0.9999 (99.99%)
• Common starting points: 0.99 (99%), 0.999 (99.9%), 0.9999 (99.99%)

**Window options:** `7d`, `14d`, `28d`, `30d`

• 28d is most common; matches many SLO frameworks
• Shorter windows (7d) react faster but have higher variance

**Alerting best practices:**

• `fastBurn`: Pages on-call (high burn rate, short window — catches rapid budget consumption)
• `slowBurn`: Creates tickets (low burn rate, long window — catches gradual degradation)

**Labels:** Use consistent label keys (team, service, environment, tier) for filtering and grouping.

**GroupByLabels** (ratio/threshold queries): Add labels like `cluster`, `service`, `endpoint` for dimens