/csp-investigation

name: observability-csp-investigation
description: >
  Investigate cloud-provider (CSP) service issues — AWS today (the reasoning applies
  to any AWS service; schema and failure signatures are deepest for RDS, Lambda, SQS,
  ALB, EC2), and in future GCP and Azure — using OTel-schema cloud metrics in Elastic.
  Use when diagnosing cloud-resource problems, whether alert-triggered or ad hoc:
  database connection/CPU/memory pressure, function error rates and throttling, queue
  backlog and consumer lag, load-balancer 5xx, instance saturation. Correlates the
  resource under investigation against its own baseline, classifies the failure mode,
  and rules out co-occurring but unrelated conditions.
compatibility: >
  Requires the `elastic` CLI (>= 0.2) with an Elasticsearch context, and cloud-provider
  metrics ingested through EDOT (the OpenTelemetry `awscloudwatch` receiver) into
  `metrics-aws.<service>.otel-*` data streams. The base floor is Elasticsearch 8.11
  or later, or Serverless; every query is ES|QL. Alert-state and SLO lookups additionally
  need a Kibana context on the same deployment. Read-only.
metadata:
  author: elastic
  version: 0.2.0
  universal: true

Cloud Provider (CSP) Investigation

Diagnose cloud-provider service issues from OTel-schema metrics and logs in Elastic. The guidelines, flow, and synthesis below are provider-agnostic and apply to every investigation. The per-provider **reference** carries the schema facts and failure-mode signatures that cannot be guessed — read it first.

This is the CSP **domain layer**: which index holds a resource's telemetry, what shape its metrics take, and how to read them into a diagnosis. It assumes the harness can already author ES|QL — it does not teach query syntax, null handling, or function usage. Keep that split: this skill tells you _what_ to query and _how to interpret_ it; writing the query itself is the harness's job.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, alert payloads, log lines, dimension names), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Every ES|QL query in this skill and in the provider references runs via `POST /_query`. Discovering which data streams and fields a deployment actually has uses `GET /_resolve/index/<pattern>` and `GET /_field_caps` (the way to learn whether a sparse counter has ever been mapped — see Guidelines). Active alerts are read with `GET kbn:/api/alerting/rules/_find`; shipped SLOs with `GET kbn:/api/observability/slos`. The [Operations](#operations) table maps each to its `elastic` CLI equivalent. Express predicates natively in ES|QL; do not use Query DSL or the `KQL` function.

Router — read the provider reference first

Identify the provider from the resource under investigation, then read its reference before writing any query (the index patterns, field shapes, and stat/aggregation rules there are unguessable, and a query written without them fails or silently corrupts its aggregates):

| Provider | Resource types | Reference | | ---------- | --------------------------------------------------------------------------------------------------- | -------------------------------------- | | AWS | Any service under `metrics-aws.*`; failure signatures documented for RDS, Lambda, SQS, ALB/ELB, EC2 | [references/aws.md](references/aws.md) | | GCP, Azure | — | not yet covered (see below) |

If the provider is unclear, determine it from the data before assuming: `GET /_resolve/index/metrics-*` lists the data streams that exist, and the provider is in their names. For a provider without a reference yet, apply the provider-agnostic guidance below, discover the schema empirically (list data streams → field mapping → probe query), cap confidence at medium, and say the reference was unavailable — never transplant another provider's schema.

The same applies _within_ a provider. For an AWS service the reference doesn't detail (DynamoDB, ECS, API Gateway, Kinesis, ElastiCache, …), the general guidance and the AWS-wide conventions still hold — the `metrics-aws.*` index family, backticked dotted/slashed field names, and per-`attributes.stat` filtering — so apply those, discover that service's specific metrics empirically, and cap confidence where no documented failure-mode signature exists. The guidelines below are service-agnostic by design; only the tuned signatures are per-service.

Guidelines

**Investigate the named resource before anything else — the one the alert fired on, or the one the request names.** Shared clusters carry many systems' telemetry (Kubernetes workload