/alibabacloud-agentloop-evaluation

name: alibabacloud-agentloop-evaluation
description: Orchestrate AgentLoop evaluation workflows through the Aliyun CLI plugin with safe previews, saved evaluator and evaluator-skill management, one-shot sample tests, trace or dataset batch runs, polling, and result inspection. Analyze evaluation quality and low-score cases from SLS. Use for natural-language requests to create or update evaluators or evaluator skills, launch or monitor evaluation tasks, inspect evaluation results, diagnose low scores, troubleshoot evaluation API calls, and simplify AgentLoop evaluation commands.

Alibaba Cloud AgentLoop Evaluation

Scenario Description

Orchestrate AgentLoop evaluation workflows through the Aliyun CLI plugin (`aliyun-cli-agentloop`). The skill converts a compact JSON specification into the required AgentLoop API sequence: discover evaluators, create/update saved evaluators, create one-shot or batch evaluation tasks, poll runs, and analyze results from SLS.

**Architecture**: `Aliyun CLI >= 3.3.3 + aliyun-cli-agentloop 0.7.0 + aliyun-cli-sls plugin + AgentLoop API (2026-05-20) + SLS evaluation_detail Logstore`

Use the bundled wrapper to convert a compact JSON specification into the required AgentLoop API sequence. Default to a dry-run; send mutations only with `--execute`.

Installation

**Pre-check: Aliyun CLI >= 3.3.3 required** > [MUST] Verify: `aliyun version` — must be >= 3.3.3. > - **First install or major upgrade:** Download, review, then execute the [setup script](references/cli-installation-guide.md#first-time-install-or-major-upgrade). Avoid `curl | bash` piping. > - **Routine update (CLI >= 3.3.5):** `aliyun upgrade` — prefer this built-in self-update over re-running the install script. > - See [references/cli-installation-guide.md](references/cli-installation-guide.md) for full installation instructions.

**Pre-check: Aliyun CLI plugin update required** > [MUST] run `aliyun configure set --auto-plugin-install true` to enable automatic plugin installation. > [MUST] run `aliyun plugin update` to ensure that any existing plugins are always up-to-date.

Verify the AgentLoop plugin:

aliyun plugin show --name aliyun-cli-agentloop
aliyun agentloop version

For result analysis, also install the SLS plugin:

aliyun plugin install --name aliyun-cli-sls

Script Dependencies

The bundled Python scripts in `scripts/` use **only the Python 3.8+ standard library** — no external packages are required. See [scripts/requirements.txt](scripts/requirements.txt) for the full declaration.

Environment Variables

| Variable | Required | Description | |----------|----------|-------------| | `SKILL_DIR` | yes | Absolute path to this skill directory (containing `SKILL.md`) | | `ALIBABA_CLOUD_PROFILE` | no | Scope the workflow to a named CLI profile without switching the default | | `SKILL_SESSION_ID` | no | 32-char hex session ID for observability; injected by the agent at runtime |

Authentication

> **Pre-check: Alibaba Cloud Credentials Required** > > **Security Rules:** > - **NEVER** read, echo, or print AK/SK values (e.g., `echo $ALIBABA_CLOUD_ACCESS_KEY_ID` is FORBIDDEN) > - **NEVER** ask the user to input AK/SK directly in the conversation or command line > - **NEVER** use `aliyun configure set` with literal credential values > - **ONLY** use `aliyun configure list` to check credential status > > ```bash > aliyun configure list > ``` > Check the output for a valid profile (AK, STS, or OAuth identity). > > **If no valid profile exists, STOP here.** > 1. Obtain credentials from [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak) > 2. Configure credentials **outside of this session** (via `aliyun configure` in terminal or environment variables in shell profile) > 3. Return and re-run after `aliyun configure list` shows a valid profile

RAM Policy

This skill requires AgentLoop and SLS permissions. See [references/ram-policies.md](references/ram-policies.md) for the full permission list.

> **[MUST] Permission Failure Handling:** When any command or API call fails due to permission errors at any point during execution, follow this process: > 1. Read `references/ram-policies.md` to get the full list of permissions required by this SKILL > 2. Use `ram-permission-diagnose` skill to guide the user through requesting the necessary permissions > 3. Pause and wait until the user confirms that the required permissions have been granted

Parameter Confirmation

> **IMPORTANT: Parameter Confirmation** — Before executing any command or API call, > ALL user-customizable parameters (e.g., AgentSpace name, Region, evaluator names, > task names, dataset names, time windows, thresholds, etc.) MUST be confirmed with the > user. Do NOT assume or use default values without explicit user approval.

| Parameter Name | Required/Optional | Description | Default Value | |---|---|---|---| | `agent_space` | Required | Target AgentSpace name | — | | `region` | Optional | Aliyun endpoint-selection region | — | | `task.name` | Optional | Evaluation task name | Auto-generated | | `task.mode` | Optional | `oneshot` or `batch` | Inferred from data_filter | | `task.data_type` | Optional | `trace` or `dataset` | `trace` | | `data_filter.max_records` | Required (batch) | Maximum records to evaluate | — | | `window.start` | Required (batch) | Backfill start time (timezone-bearing ISO-8601) | — | | `window.end` | Required (batch) | Backfill end time (timezone-bearing ISO-8601) | — | | `threshold` | Optional | Low-score threshold for analysis | `0.5` | | `max_cases` | Optional | Maximum low-score cases to return | `50` |

Mutation Confirmation Protocol

> **CRITICAL: All cloud mutation operations require explicit user confirmation before execution.** > > This skill can create, modify, and delete cloud resources. To prevent unintended > changes, every mutation MUST follow the three-step protocol below. No exceptions.

Mutation operations requiring confirmation