/agent-platform-eval-flywheel

name: agent-platform-eval-flywheel
metadata:
  version: "1.0.4"
  category: AiAndMachineLearning
description: >-
  Measures and improves the quality of AI models and agents on Google Cloud
  using the Eval Quality Flywheel methodology. Use when generating synthetic
  user scenarios, evaluating an agent or model, building an eval dataset,
  picking or writing evaluation metrics, analyzing failures, comparing results
  before and after a fix, or when guidance is needed on Agent Platform eval
  methodology — including dataset schema, LLM-as-judge scoring, and common
  failure causes. For fine-tuning, use agent-platform-tuning. For general
  production deployment, use agent-platform-deploy.

Agent Platform Eval Flywheel Skill

Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK (`google.genai` / `agentplatform`).

Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you MUST read [Usage Attribution](references/attribution.md). For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix `gcloud` with `CLOUDSDK_METRICS_REQUEST_ATTRIBUTION` and append the attribution suffix to `User-Agent` headers:

CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-eval-flywheel)"
User-Agent: <existing_user_agent> agent-skills/1.0 (skill:agent-platform-eval-flywheel)

When to use this skill

• Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation

SDK (`client.evals.evaluate()`).

• Creating evaluation datasets from session traces, pandas DataFrames, or

synthetic generation.

• Selecting, configuring, or writing custom evaluation metrics.
• Analyzing rubric verdicts, loss patterns, and clustering failures.
• Suggesting concrete code/prompt improvements based on eval results.
• Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a

**Model-as-a-Service (MaaS)** model by ID — including deploying the model first if needed. For this case, follow [references/deployment.md](references/deployment.md) and use the `endpoint_evaluation.py` / `maas_evaluation.py` scripts.

Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`, `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`)

• **Rule**: No confirmation needed. You may execute these helper scripts

immediately to inspect data, validate schemas, parse traces, or compare evaluation results. 2. **Tier M: Evaluation & Generation Runs (`client.evals.run_inference`, `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`, `client.evals.generate_loss_clusters`)**

• **Rule**: For predictable user approval over dataset reads, result

writes (`dest`), and remote compute/API usage, every call to `client.evals.evaluate()`, `client.evals.run_inference()`, `client.evals.generate_conversation_scenarios()`, or `client.evals.generate_loss_clusters()` — whether using remote metrics (LLM/rubric metrics or sandboxed `types.CodeExecutionMetric`) or local in-process metrics (`types.Metric` with a Python callable) — requires presenting a dry-run confirmation preview showing the proposed evaluation call and parameters and waiting for explicit 'Yes'/'No' user approval (do not quote remote compute charges when only local in-process `types.Metric` callables are used without `dest` or model inference).

• **Confirmation for EVERY evaluation run**: Every evaluation,

re-evaluation, metric update, parameter change, or synthetic scenario generation requires its own dry-run preview and interactive confirmation. Never execute a second evaluation, comparison pass, or modified evaluation without presenting a new confirmation preview and obtaining user approval.

• **Same-turn restriction**: Do not run the evaluation in the same turn as

presenting the confirmation prompt. End your turn after asking and wait for the user's reply; only execute after explicit 'Yes' / approval. Printing a preview and then calling the tool before the user can answer does not count as obtaining confirmation.

• **No Pre-Execution of Evaluation Calls**: NEVER execute

`client.evals.evaluate()`, `client.evals.run_inference()`, `client.evals.generate_conversation_scenarios()`, or run any script invoking these SDK methods before user confirmation. In the initial turn, you may write the evaluation script to disk, but you MUST present the dry-run confirmation preview and obtain explicit user confirmation before running `client.evals.evaluate()`.

• **Immediate Execution Upon Approval**: Once the user explicitly approves

(e.g., 'Yes', 'Approved', 'Go ahead', 'Proceed'), proceed directly to executing the previewed evaluation script via `run_command` and report the results. Do not conclude the turn without executing the approved action.

Setup

The scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`), `google-genai`, `pandas`, and `requests`. Do **not** create a virtual environment — it starts empty and hides packages the environment already provides, forcing a redundant install. Probe, and install only what is missing:

python3 -c "import vertexai, google.genai, pandas, requests" \
  || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'

The version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a redirect and silently writes an empty file instead of constraining the install.

Ne