/elasticsearch-cluster-health

name: elasticsearch-cluster-health
description: >
  Diagnose a non-green Elasticsearch cluster and surface the single most likely cause
  with remediation. Use when an operator reports yellow or red status, unassigned
  shards, allocation failures, or wants read-only triage before deeper investigation.
  Teaches replica-vs-primary impact, allocation decider classification, and data-loss
  awareness.
metadata:
  author: elastic
  version: 0.1.0
  universal: true
compatibility: Elasticsearch 8.x or 9.x, self-managed or Elastic Cloud Hosted; not
  applicable to Elastic Cloud Serverless, where cluster, shard, and allocation APIs
  are managed internally. Requires the `elastic` CLI ≥ 0.2 with `stack es` support.

Diagnose Cluster Health

Triage a non-green Elasticsearch cluster read-only: localize the problem, classify the allocation decider, and report the single most likely cause with remediation. Never mutate cluster state — surface findings and let the operator act.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Process

1. **Read the overall status.** Call `GET /_cluster/health`. The `status` field is the verdict:

• `green` — every primary and replica is assigned. Report healthy and stop.
• `yellow` — every **primary** is assigned but at least one **replica** is not. Data remains readable; redundancy is

degraded. This is **not** data loss.

• `red` — at least one **primary** is unassigned. Data for that shard is **unavailable**; treat as urgent.

Also read `unassigned_shards`, `initializing_shards`, and `relocating_shards`. The decision: continue only when status is yellow or red. If `initializing_shards > 0` and `unassigned_shards == 0`, the cluster is recovering on its own — call `GET /_cat/recovery` to confirm progress, wait, and re-check `GET /_cluster/health` before escalating.

Data needed: cluster-wide `status` and shard counters.

2. **Localize the problem to one index.** Call `GET /_cluster/health?level=indices` and pick the index that drives the cluster-wide status:

• Any **red** index outranks every yellow index.
• Among reds or yellows, prefer the index with the most `unassigned_shards`.
• A red **system** index (`.security`, `.kibana*`, `.fleet-*`) outranks application indices because the rest of the

stack depends on it.

Optionally call `GET /_cat/shards/{index}?h=index,shard,prirep,state,unassigned.reason` to list every unassigned shard on that index and see whether failures are primaries (`prirep=p`) or replicas (`prirep=r`).

The decision: focus the next steps on exactly **one** index — the one whose recovery unblocks the cluster.

Data needed: per-index `status` and `unassigned_shards`; shard role (primary vs replica) when available.

3. **Separate trigger from root cause.** Call `POST /_cluster/allocation/explain` with no body so Elasticsearch selects an unassigned shard, or target the worst shard explicitly:

   { "index": "<index>", "shard": <id>, "primary": <true|false> }

Read these fields in order:

• `primary` — `false` means a **replica** is unassigned (typical yellow); `true` means a **primary** is unassigned

(typical red).

• `can_allocate` — top-level allocation verdict (`no`, `yes`, `throttled`, `no_valid_shard_copy`, …).
• `unassigned_info.reason` — what **triggered** reassignment (e.g. `NODE_LEFT`, `INDEX_CREATED`). This is **not** the

root cause when `can_allocate` is `no`; it only explains why the shard became unassigned.

• `allocate_explanation` — human-readable summary; quote it verbatim in the report.
• `node_allocation_decisions[].deciders[]` — per-node decider results. Find deciders with `decision: "NO"`; the

**decider name** (e.g. `disk_threshold`, `filter`, `awareness`) is the root cause class.

The decision:

• **Yellow + `primary: false`** — impact is limited to replica redundancy; no data loss. Continue to step 4 to name

the blocking decider (do **not** stop at `NODE_LEFT`).

• **Red + `primary: true`** — data for that shard is missing. Continue to step 4; if `can_allocate` is

`no_valid_shard_copy`, treat as potential **data loss** immediately.

Data needed: allocation-explain response for one representative unassigned shard on the chosen index.

4. **Classify the decider.** Map the blocking signal to a cause class. Prefer the decider with `decision: "NO"` over the `unassigned_info.reason` trigger.

| Signal | Cause class | Typical remediation (operator applies) | | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `decider: disk_threshold`, `decision: NO` | Disk high/low watermark exceeded | Free disk on the named node, add data-node capacity, or adjust `cluster.routing.allocation.disk.watermark.*` a