/alibabacloud-analyticdb-spark-application-analysis-helper

name: alibabacloud-analyticdb-spark-application-analysis-helper
description: >
  Analyze and troubleshoot Alibaba Cloud AnalyticDB (ADB) Spark applications—execution monitoring, failure diagnosis, performance anomaly detection, and cross-application comparison.
  Use this Skill when users need to:
  1. Check the execution status of AnalyticsDB Spark applications (running, succeeded, failed, etc.).
  2. Analyze root causes of failed AnalyticsDB Spark applications (e.g., OOM, data skew, dependency errors, permission issues).
  3. Identify Spark applications with abnormal execution duration by statistical analysis.
  4. Compare similar Spark applications to pinpoint performance anomalies or detect SQL execution plan differences.
  Also applicable for "why did my Spark job fail", "which Spark app is running slow", "compare two Spark runs", etc.
license: MIT
compatibility: >
  Requires Alibaba Cloud CLI (aliyun >= 3.3.3), with AccessKey or STS Token configured.
  Verify credentials via `aliyun configure list`.
metadata:
  domain: aiops
  owner: adb-spark-team
  contact: shixing.gm@alibaba-inc.com
  required_permissions: references/ram-policies.md

Alibaba Cloud AnalyticsDB Spark Application Analysis & Troubleshooting

Analyze and troubleshoot AnalyticsDB Spark applications via `aliyun` CLI and local analysis scripts. You are a Spark-savvy SRE—not just an API caller, but someone who knows how to diagnose failures, detect performance anomalies, and compare execution plans across similar applications.

Authentication

Reuse the configured `aliyun` CLI profile. Switch accounts with `--profile <name>`, check configuration with `aliyun configure list`.

Before execution, read [ram-policies.md](references/ram-policies.md) if you need to confirm the minimum RAM authorization scope.

Installation

**Pre-check: Aliyun CLI >= 3.3.3 required** > Run `aliyun version` to verify >= 3.3.3. If not installed or version too low, refer to `https://www.alibabacloud.com/help/en/cli/` find the latest version and installation instructions for your OS. > or see `references/cli-installation-guide.md` for installation instructions.

**Pre-check: Aliyun CLI plugin update required** > [MUST] run `aliyun configure set --auto-plugin-install true` to enable automatic plugin installation. > [MUST] run `aliyun plugin update` to ensure that any existing plugins are always up-to-date.

**[MUST] AI-Mode Setup** — Before executing any CLI commands, enable AI-Mode and set User-Agent:

aliyun configure ai-mode enable
aliyun configure ai-mode set-user-agent --user-agent "AlibabaCloud-Agent-Skills/alibabacloud-analyticdb-spark-application-analysis-helper"

After all workflow operations are complete, disable AI-Mode:

aliyun configure ai-mode disable

**[MUST] CLI User-Agent** — Every `aliyun` CLI command invocation must include: `--user-agent AlibabaCloud-Agent-Skills/alibabacloud-analyticdb-spark-application-analysis-helper`

**[MUST] ossutil Pre-check** — Required before any OSS read operation (event log download, full log analysis). The Skill MUST verify both availability and configuration:

1. **Availability check** — run `aliyun ossutil version`. If the command exits non-zero or reports `command not found`, STOP and report to the user: > "`aliyun ossutil` is not available. Install it via `aliyun plugin install --name ossutil`, then re-run the workflow. OSS event log and full log analysis cannot proceed without ossutil."

2. **Configuration check** — `aliyun ossutil` inherits credentials from the active `aliyun` CLI profile (default credential chain). **DO NOT** explicitly handle AK/SK — never run `aliyun ossutil config -i <AK> -k <SK>` or any equivalent command that writes credentials. Use `aliyun ossutil config get` only for read-only inspection. To verify the credential chain is healthy, run `aliyun configure list` and confirm an active profile (AK / STS / RAM role / ECS role) is present. The endpoint should be supplied per-command via `-e oss-<region>.aliyuncs.com` matching the target cluster region. If runtime authentication fails, STOP and report to the user: > "`aliyun ossutil` cannot authenticate via the default credential chain. Verify the active profile via `aliyun configure list` and confirm it has `oss:GetObject` / `oss:ListObjects` on the log bucket. DO NOT pass AK/SK directly to ossutil — fix the underlying `aliyun` profile or RAM policy instead."

3. **DO NOT proceed** to Event Log download or OSS log analysis until both checks pass. **DO NOT** fall back to fabricating data — see Category 3 in the Safety Constraints section.

Execution Principles

1. **Check documentation before acting**: Before calling any API, consult `references/api-reference.md` to confirm parameter names and formats. Never guess parameter names from memory. 2. **Return to documentation on errors — MANDATORY**: When any API call fails, STOP. Do NOT retry with variations. Go directly to `references/api-reference.md`, find the exact error code, read the correct parameter specification, then retry ONCE with the corrected command. Blind retry loops are prohibited.

Input Format Validation [MANDATORY]

After collecting the Spark Application ID and DBClusterId from the user, **format validation MUST be performed first**. Only proceed to subsequent API calls once validation passes. If the format does not match, prompt the user to check the input and provide a correct-format example.

DBClusterId Format Rules

| Rule | Description | |--------|------| | Prefix | Must start with `amv-` or `am-` | | Trailing characters | Fixed-length lowercase letters + digits (16 characters) | | Regex | `^(amv|am)-[a-z0-9]{16}$` | | Example | `amv-2zeu6ug6wn705j19` |

Spark Application ID Format Rules

| Rule | Description | |--------|------| | Prefix | Must start with `s` | | Timestamp segment | Minute-precision timestamp in `YYYYMMDDHHmm` format (12 digits) | | Region abbreviation | 2-3 lowercase letters, e.g.