/alibabacloud-lingjun-node-diagnose

name: alibabacloud-lingjun-node-diagnose
description: |
  bash prefix: export LJ_SKILL_DIR="${LJ_SKILL_DIR:-$HOME/.qoder/skills/alibabacloud-lingjun-node-diagnose}" && source "$LJ_SKILL_DIR/lib/lj_init.sh"
  i18n: detect language each turn (CJK ratio ≥ 0.30 → LJ_LANG=zh, else en); inject LJ_LANG=zh|en prefix before source.
  Intelligent diagnosis for Alibaba Cloud Lingjun (regular & hyper) compute nodes: submit diagnostic tasks (BasicCheck / NodeHardwareCheck / CheckByAiJobLogs), produce structured diagnostic reports, propose repair plans (reboot / reimage / stop / report-node-status), track fault reports (list-fault-reports / describe-fault-report / stop-node-diagnostic). Read-only: list-clusters / list-cluster-nodes / list-cluster-hyper-nodes / describe-node / describe-hyper-node / list-syslogs / list-diagnostic-results.
  Triggers: "lingjun diagnose", "灵骏诊断", "灵骏排障", "GPU 故障", "硬件故障", "节点异常", "诊断节点", "重启节点", "重装节点", "灵骏修复", "报障", "终止报障", "syslog", "NodeHardwareCheck", "CheckByAiJobLogs", "BasicCheck", "fault report", "stop fault diagnosis"

Alibaba Cloud Lingjun Node Diagnose & Repair

Scenario Description

Alibaba Cloud Lingjun runs AI workloads on rack-level GPU bare-metal nodes. Failures (GPU/HBM ECC, NIC link flap, NCCL hang, kernel panic, RDMA degradation, AI-job log anomalies) require a closed-loop "diagnose → report → repair" workflow. This skill covers exactly that closed loop using the **eflo-controller (v2022-12-15)** OpenAPI namespace shared with the `alibabacloud-lingjun-cluster-scaling` skill — three diagnostic CLIs + four repair CLIs + three fault-report CLIs + supporting read-only helpers — across **8 features**:

1. **Resource Locator** — `list-clusters` / `describe-cluster` / `list-cluster-nodes` / `list-cluster-hyper-nodes` / `describe-node` / `describe-hyper-node` to anchor the target Cluster + (Hyper)Node before any diagnostic submission. 2. **Submit Diagnostic Task** — `create-diagnostic-task` with one of three `DiagnosticType` values (`BasicCheck` / `NodeHardwareCheck` / `CheckByAiJobLogs`) — enum re-verified server-side 2026-08-19 (`NetConfigCheck` / `NetRuntimeCheck` deprecated, never submit). 3. **Query Diagnostic Result (single)** — `describe-diagnostic-result` returns the per-node check items + verdict + remediation hints. 4. **List Diagnostic History** — `list-diagnostic-results` browses prior diagnostic tasks (paginated, optional `--diag-type` filter). 5. **Produce Diagnostic Report** — Render a Markdown report combining (1) target identity, (2) diagnostic verdict, (3) per-check-item table, (4) supporting evidence (syslog excerpts, hardware counters). 6. **Produce Repair Plan** — Map the diagnostic verdict to one of `reboot-nodes` / `reimage-nodes` / `stop-nodes` / `report-node-status` (or escalate to the cluster-scaling skill's `shrink-cluster`/`delete-node` for permanent removal); output a `safe_mutate`-compatible HITL plan. 7. **Auxiliary Telemetry** — `list-syslogs` (kernel/system log excerpts), per-node hardware counters from `describe-node` / `describe-hyper-node` to enrich the report. 8. **Fault Report Tracking** — after `report-node-status` (fault declaration), track the deep-diagnosis lifecycle via `list-fault-reports` / `describe-fault-report`, stop an in-progress fault diagnosis via `stop-node-diagnostic`, and approve a platform-raised maintenance proposal via `approve-operation` (both mutating, `safe_mutate` two-phase).

**Key Resources**: Cluster → (Node Groups) → Compute Nodes — regular `NodeId` (e.g., `e01-cn-...`) or rack-level `HyperNodeId` (e.g., `hn-cn-...`). Diagnostic tasks operate on `NodeId` or `HyperNodeId`.

**Supported Regions**: Use `safe_aliyun aliyun eflo-controller describe-regions --endpoint eflo-controller.cn-hangzhou.aliyuncs.com --region cn-hangzhou` (`describe-regions` is the discovery seed; see [endpoint-routing.md](references/endpoint-routing.md) §2 sole exception) or fall back to [supported-regions.md](references/supported-regions.md).

---

Installation

Verify `aliyun version >= 3.3.3`; otherwise:

curl -fsSL --connect-timeout 10 --max-time 120 https://aliyuncli.alicdn.com/setup.sh | bash
aliyun version

This skill uses **one** Alibaba Cloud OpenAPI namespace — only `eflo-controller` plugin is required:

| Plugin | CLI Namespace | Used For | |---|---|---| | `eflo-controller` | `aliyun eflo-controller ...` | Diagnostic submit/query/list, repair (reboot/reimage/stop/report-node-status), fault-report tracking (list/describe/stop-node-diagnostic), and all read-only locators |

aliyun configure set --auto-plugin-install true
aliyun plugin install --name eflo-controller
aliyun plugin update

See [cli-installation-guide.md](references/cli-installation-guide.md) for detailed instructions and verification.

---

Authentication

Verify credentials via `aliyun configure list` only. **Never** run `aliyun configure get` / `configure show` (they print plaintext secrets) and **never** echo or display AccessKey values; mask any credential-bearing output (e.g. `aliyun configure list | sed -E 's/(LTAI[A-Za-z0-9]{4})[A-Za-z0-9]+/\1****/g'`). If missing, guide users to the [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak) outside this session.

---

RAM Permissions

Namespace coverage: `eflo:*` (read-only diagnostic + node-state mutating). Split into 4 permission sets:

• **Read-Only Locator (8)**: `eflo:ListClusters` / `eflo:DescribeCluster` / `eflo:ListClusterNodes` / `eflo:ListClusterHyperNodes` / `eflo:DescribeNode` / `eflo:DescribeHyperNode` / `eflo:DescribeRegions` / `eflo:ListSyslogs`.
• **Diagnostic Read+Submit (3)**: `eflo:CreateDiagnosticTask` / `eflo:DescribeDiagnosticResult` / `eflo:ListDiagnosticResults`.
• **Repair Mutating (4)**: `eflo:RebootNodes` / `eflo:ReimageNodes` / `eflo:StopNodes` / `eflo:ReportNodeStatus`.
• **Fault Report Tracking (4)**: `eflo:DescribeFaultReport` / `eflo:ListFaultReports` (read-only) + `eflo:StopNodeDiagnostic` / `eflo:ApproveOperati