/sre-triage

name: observability-sre-triage
description: >
  Triage a degraded or suspect service end to end: read SLO status and burn rate,
  check active alerting rules and ML anomalies, measure throughput, latency, and error
  rate, assess dependency health and infrastructure saturation, and funnel logs down
  to the failures that explain it. Use when someone asks whether a service is healthy,
  why it is slow or erroring, what is in its logs, or which attribute distinguishes
  the requests that are failing. Also use when someone asks for the query behind any
  of those signals — throughput, latency percentiles, error rate, dependency health,
  or log volume — over APM/OTel traces, metrics, or logs.
compatibility: >
  Requires the `elastic` CLI (>= 0.2) with Elasticsearch and Kibana contexts on the
  same cluster. Base floor is Elasticsearch 8.11+ or Serverless. Three ES|QL features
  need more, each with a fallback at its point of use: `FORK` (Stack GA 9.4), `CATEGORIZE`
  (Stack GA 9.1, Platinum licence) and `TS` (Stack GA 9.4); all are GA on Serverless.
  Reads APM/OTel traces, metrics and logs, the Kibana SLO and Alerting APIs, and the
  Elasticsearch ML APIs. Read-only.
metadata:
  author: elastic
  version: 0.5.1
  universal: true

SRE Service Triage

Decide whether a service is healthy, degraded, or unhealthy, and say why. Triage is a hierarchy, not a checklist: SLOs and alerts define whether the service is failing its contract, trace-derived golden signals describe how it is failing, dependencies and infrastructure explain where the failure comes from, and logs supply the sentence you put in the incident channel. Work down the hierarchy until the evidence supports a verdict, then stop.

For authoring and tuning SLO definitions, burn-rate rules, and alert thresholds, use the **observability-service-reliability** skill. This skill only reads that state. For Kubernetes workload, node, or control-plane diagnosis — restart loops, OOM kill confirmation, node pressure, admission rejections, stuck rollouts — hand off to the **observability-k8s-investigation** skill. This skill checks whether a Kubernetes-hosted service is saturated; it does not diagnose why the pod or the node behind it is failing.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Everything here is expressed in ES|QL (`POST /_query`) or the Kibana Observability APIs. Do not use Query DSL, and do not use the ES|QL `KQL` search function — express predicates natively (`WHERE service.name == "checkout"`).

Jobs to be done

• Answer "is service X healthy?" with a verdict and the evidence behind it
• Answer "why is service X slow / erroring / quiet?" by localizing the change to the service, a dependency, or its

infrastructure

• Read SLO status, burn rate, and remaining error budget during an incident
• Determine which alerting rules currently apply to a service, including all-services rules
• Funnel a noisy log stream down to the failures that explain the degradation
• Identify which attribute (version, host, pod, region, route) distinguishes the failing or slow subpopulation
• Distinguish a healthy service from a service with no telemetry

Output discipline

Applies to every response produced under this skill.

• **Commit to the best-supported conclusion.** When the evidence points one way, say so. Do not downgrade confidence to

sound cautious — hedging on unambiguous evidence is a defect, not humility.

• **Commit to a verdict**: healthy, degraded, or unhealthy, followed by the reason. A triage answer that does not name

one of the three has not done the job.

• **State confidence once**, in the conclusion. Do not restate it per bullet.
• **Do not speculate past the evidence.** If the telemetry did not show a cause, it does not go in the answer. Name what

is unknown and stop. Never offer a mechanism ("probably a GC pause", "likely a noisy neighbor") that no signal measured.

• **Report absence as absence.** Zero rows means the data is missing or not collected; it never means the underlying

condition is healthy. "No dependency metrics" is not "dependencies are fine".

• **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered, no

summarizing the summary.

• **End on the finding.** No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a

recommendations list, phrased as recommendations, not as questions.

Signal hierarchy

Signals disagree constantly. This ordering decides which one wins.

| Rank | Signal | Authority | | ---- | -------