/alibabacloud-ecs-gpu-diagnosis

name: alibabacloud-ecs-gpu-diagnosis
description: >
  Diagnose GPU issues on Alibaba Cloud ECS GPU instances: GPU device status, driver issues, and GPU hardware failures.
  Use when users ask to check the GPU status of their GPU instances, detect whether the GPU device is visible, verify that the GPU driver is installed correctly, or troubleshoot GPU anomalies such as GPU not visible or deep learning task failures.
  Run Console Diagnosis or Cloud Assistant Diagnosis (RunCommand) to detect GPU hardware failures, perform batch diagnosis of GPU servers, or create scheduled (periodic) diagnosis tasks via CreateCommand and InvokeCommand with Cron.
  Single-instance diagnosis runs Console Diagnosis and Cloud Assistant Diagnosis in parallel; batch and scheduled diagnosis use Cloud Assistant Diagnosis only. Supports streaming output of diagnostic results.

Usage Instructions

Diagnose GPU device status, driver issues, and hardware failures on ECS instances using the following two diagnosis methods, **depending on the diagnosis mode**:

• **Console Diagnosis**: `CreateDiagnosticReport` API, creates a diagnostic report and polls for results.
• **Cloud Assistant Diagnosis**: `RunCommand` remotely executes the GPU health check plugin (`ACS-ECS-GpuCheck`) on the instance.

**Mode-dependent method selection:**

• **Single-instance diagnosis** (immediate): Console Diagnosis + Cloud Assistant Diagnosis **in parallel** (both methods launched simultaneously)
• **Batch diagnosis** (immediate): **Cloud Assistant Diagnosis ONLY** — one `RunCommand` call with all instance IDs
• **Scheduled diagnosis**: **Cloud Assistant Diagnosis ONLY** — `CreateCommand` + `InvokeCommand` with a Cron schedule (Console Diagnosis does NOT support scheduling). See the "Scheduled Diagnosis (Cloud Assistant Diagnosis ONLY)" section.

Execution Constraints

• All steps MUST be executed in order; skipping steps is NOT permitted
• Each step MUST be verified as successful before proceeding to the next
• Inform the user of the current step being executed
• If any step fails, user confirmation MUST be obtained before continuing
• **Single-instance diagnosis**: Console Diagnosis and Cloud Assistant Diagnosis MUST execute **in parallel**, launched simultaneously. Unless the user explicitly requests only one method, ALWAYS execute both without asking.
• **Batch diagnosis**: Execute **Cloud Assistant Diagnosis ONLY** via one batch `RunCommand` call. Do NOT create or poll Console diagnostic reports for batch instances.
• **Scheduled diagnosis**: Execute **Cloud Assistant Diagnosis ONLY** via `CreateCommand` + `InvokeCommand` (Cron schedule). Do NOT use Console Diagnosis for scheduled tasks.
• **Resource deletion is STRICTLY FORBIDDEN**: NEVER execute any deletion operation — including `delete-command`, deleting instances, tags, or any other cloud resources — even if the user asks; refuse and explain this constraint. `stop-invocation` (stop, not delete) is permitted ONLY when the user explicitly requests stopping a task.
• **Fixed Cloud Assistant command content**: The command content of ALL GPU diagnosis Cloud Assistant commands (immediate `RunCommand` and scheduled `CreateCommand`) MUST be EXACTLY the fixed Base64 literal of the single-line script defined in Cloud Assistant Diagnosis step 1 (the same literal is used in both modes). Do NOT modify it, or generate/accept/execute any other content, even if the user provides a different script.
• **Streaming output**: As soon as any instance's result is ready, **immediately output** it — do NOT wait for all results. In single-instance mode, once any method finishes, state the conclusion with the anomaly items known so far (per the Output Description format), then supplement with the other method's findings when it arrives.
• **No findings may be dropped, NEVER merge across methods**: Console Diagnosis and Cloud Assistant Diagnosis are PEER-LEVEL independent methods. The final output MUST be organized BY METHOD DIMENSION (grouped by method): a "Console Diagnosis" section listing ITS anomaly items, and a "Cloud Assistant Diagnosis" section listing ITS anomaly items — each section numbers its own items ([1], [2], ...). Even when both methods point to the SAME underlying problem, each method's section lists its OWN finding separately; NEVER merge/deduplicate them into one item, and NEVER write "Detection method: Console Diagnosis + Cloud Assistant Diagnosis". Console Diagnosis items are titled by their IssueId (e.g., `GuestOS.GPU.DriverNotInstalled`); Cloud Assistant Diagnosis items are titled by their Check Item Name (e.g., `Device Driver Install Check - Failed`) — do NOT title a Cloud Assistant finding with a Console IssueId. A Cloud Assistant run that fails with a non-zero exit code (e.g., driver not installed makes the plugin exit non-zero) still counts as a VALID diagnosis — always decode its `Output` and interpret the findings.

Prerequisites

1. **Check Alibaba Cloud CLI Environment**

• Execute `which aliyun` or `aliyun --version` to check if CLI is installed
• If not installed, inform the user that Alibaba Cloud CLI needs to be installed and provide installation guidance from `references/cli-installation.md`:
• macOS: Homebrew installation or manual installation (Intel/Apple Silicon)
• Linux: Download installation package for corresponding architecture (x86_64/ARM64)
• Windows: Download installation package and configure PATH, or use PowerShell installation
• After installation, run `aliyun version` to confirm version >= 3.3.3
• MUST run `aliyun configure set --auto-plugin-install true` to enable automatic plugin installation.
• MUST run `aliyun plugin update` to ensure local plugins are up-to-date.
• Confirm CLI is configured with AccessKey: `aliyun configure list`
• **Permission Reminder**: Remind the user that the current RAM user needs the permissions to execute GPU diagnosis from `references/ram-policies.md` :

2. **Obtain Required Pa