/alibabacloud-elasticsearch-instance-diagnose

name: alibabacloud-elasticsearch-instance-diagnose
description: |
  Alibaba Cloud Elasticsearch instance diagnosis skill. Use for cluster health checks, troubleshooting, and performance analysis on Elasticsearch instances.

  Triggers (English): Elasticsearch diagnosis, ES instance issues, slow search, write failures, cluster Red/Yellow, unassigned shards, node disconnected, load imbalance, thread pool 429, JVM/OOM/circuit breaker, disk watermark / read-only index, instance activating / change stuck, service avalanche / all shards failed.

  触发词（中文）: ES诊断、阿里云ES、Elasticsearch诊断、ES集群/实例故障排查、ES健康检查、集群红灯/变红/黄灯/变黄、集群异常、分片未分配、主分片未分配、节点掉线/离线、负载不均衡、搜索/查询变慢、慢查询、写入失败/变慢/拒绝、线程池打满、HTTP 429、内存过高、OOM、断路器、磁盘满/水位、索引只读、实例激活中/activating、变更卡住/未完成、雪崩、服务不可用、all shards failed。

Alibaba Cloud Elasticsearch Instance Diagnosis

Collect signals from **Alibaba Cloud OpenAPI (control plane)** and the **Elasticsearch REST API (data plane)**, combine them with the SOP knowledge base under `references/`, and produce **root-cause analysis**, an **evidence chain**, **prioritized remediation guidance**, and—when multiple dimensions fire—a **recency-ordered incident timeline** (severity vs time in window; see **Timeline and recency (MUST)** in §5 Step 4).

**Architecture**: Alibaba Cloud Elasticsearch OpenAPI + Alibaba CloudMonitor (CMS) + Elasticsearch REST API + diagnostic SOPs

**Closure**: If MUST applies and `ES_*` is set, finish authenticated ES API evidence before the final report (see **Feasibility order** in §5).

---

1. Prerequisites

1.1 Aliyun CLI

> **Pre-check: Aliyun CLI >= 3.3.1 required** (for RAM permission checks and OpenAPI CLI fallback) > Run `aliyun version` to verify the version is >= 3.3.1. If the CLI is missing or too old, see `references/cli-installation-guide.md`. > After installation, run `aliyun configure set --auto-plugin-install true` to enable automatic plugin installation (**do not** pass plaintext AccessKey pairs on this command line; see §1.2).

1.2 Alibaba Cloud account authentication and security (MUST)

> **Security rules (mandatory):** > - **NEVER** read, echo, or print AccessKey ID or AccessKey Secret values. > - **NEVER** prompt or ask the user to paste plaintext AccessKeys in the conversation. > - **NEVER** embed AccessKeys in scripts, CLI arguments, or `curl` URLs. > - **NEVER** use `aliyun configure set` (or similar) to pass **literal** AccessKey ID/Secret on the command line. > - **NEVER** accept AccessKeys that the user pastes into the chat, even if offered voluntarily. > - **ONLY** use configured CLI profiles (`aliyun configure`) or environment variables such as `ALIBABA_CLOUD_ACCESS_KEY_ID` / `ALIBABA_CLOUD_ACCESS_KEY_SECRET` that the user has set **in their local shell** (the agent **must not** echo those values in the session).

> **⚠️ If the user provides AccessKeys in the chat (e.g. “my AK is xxx”)** > > 1. **Stop immediately**: do not run any Alibaba Cloud command that requires credentials. > 2. **Decline politely** and give **only** the names of approved configuration methods (**do not** repeat any secret the user may have leaked): > - Recommended: run `aliyun configure` in a local terminal and enter credentials when prompted; credentials are stored in the local profile file. > - Alternatively: set `ALIBABA_CLOUD_ACCESS_KEY_ID` / `ALIBABA_CLOUD_ACCESS_KEY_SECRET` in the local shell (the user types values **only in the terminal**, not in chat). > 3. Resume the diagnosis request only after credentials are configured correctly.

> **Verify credentials without exposing secrets:** > > ```bash > aliyun configure list > aliyun --profile <profile_name> sts get-caller-identity > ``` > > **Credential policy:** > 1. Prefer an `aliyun configure` profile (default or `--profile`). > 2. If there is no valid identity (`configure list` / `get-caller-identity` fails), **STOP** and guide the user to configure locally; **do not** guess or fabricate credentials. > 3. Never pass plaintext AccessKeys through the conversation.

1.3 Elasticsearch direct-connect credential boundary

> - **NEVER** ask the user to paste `ES_PASSWORD` in chat; **NEVER** echo, print, or log the password; **NEVER** copy a password from chat into commands, hooks, or repo files. > - Shell expansion for `curl -u "$ES_USERNAME:$ES_PASSWORD"` (or equivalent) is **allowed** when vars are **pre-exported in the user’s local shell**; **NEVER** put the secret as a literal in chat, scripts checked into repos, or command output. > - If the user tries to send a password in chat: **STOP** as well and ask them to set `ES_PASSWORD` only locally via `export` (see §2.2).

---

2. Environment setup

2.1 Control plane OpenAPI (via Aliyun CLI)

All control-plane and CMS data collection for this skill uses the **Aliyun CLI**.

> **[MUST] `elasticsearch` / `cms` — plugin-mode shell only (avoid legacy CLI)** > Whenever the agent emits **executable** `aliyun` lines (chat, reproducibility exports, or copy-paste steps), use **plugin subcommands** (lowercase-hyphenated) and **kebab-case** flags — the same shape as `scripts/openapi_cli_collect.py` and [references/verification-method.md](references/verification-method.md). > - **Do not** use legacy POP-style invocations: a **PascalCase verb** immediately after `elasticsearch` or `cms` on the same `aliyun` line (the old “action name = subcommand” style), or CamelCase flags like `--InstanceId`, `--Namespace`, `--StartTime` in **new** commands. Use plugin verbs only (`describe-instance`, `describe-metric-list`, …). > - **Naming split:** `DescribeInstance`, `ListSearchLog`, `DescribeMetricList`, etc. are **OpenAPI action names** (PascalCase — docs, RAM, console). The token **after** `aliyun elasticsearch` or `aliyun cms` in a shell must be the **CLI plugin** name (`describe-instance`, `list-search-log`, `describe-metric-list`, …). > - **Prefer** `python3 scripts/check_es_instance_health.py` for the standard control-plane + CMS bundle so subprocess calls