/agent-platform-eval-flywheel

name: agent-platform-eval-flywheel
metadata:
  category: AiAndMachineLearning
description: >-
  Measures and improves the quality of AI models and agents on Google Cloud
  using the Eval Quality Flywheel methodology. Use when evaluating an agent or
  model, building an eval dataset, picking or writing evaluation metrics,
  analyzing failures, comparing results before and after a fix, or when
  guidance is needed on Agent Platform eval methodology — including
  dataset schema, LLM-as-judge scoring, and common failure causes. For
  fine-tuning, use agent-platform-tuning. For general production deployment,
  use agent-platform-deploy.

Agent Platform Eval Flywheel Skill

Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK (`google.genai` / `agentplatform`).

When to use this skill

• Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation

SDK (`client.evals.evaluate()`).

• Creating evaluation datasets from session traces, pandas DataFrames, or

synthetic generation.

• Selecting, configuring, or writing custom evaluation metrics.
• Analyzing rubric verdicts, loss patterns, and clustering failures.
• Suggesting concrete code/prompt improvements based on eval results.
• Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a

**Model-as-a-Service (MaaS)** model by ID — including deploying the model first if needed. For this case, follow [references/deployment.md](references/deployment.md) and use the `endpoint_evaluation.py` / `maas_evaluation.py` scripts.

Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`, `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`)

• **Rule**: No confirmation needed. You may execute these helper scripts

immediately to inspect data, validate schemas, parse traces, or compare evaluation results. 2. **Tier M: Read-only with Compute Costs (`client.evals.run_inference`, `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`, `client.evals.generate_loss_clusters`)**

• **Rule**: These operations invoke LLMs or remote evaluation services

that consume compute resources and incur costs. This requires **interactive confirmation** with 'Yes'/'No' options. Once granted once, you do not have to prompt for future evaluation.

Setup

The scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`), `google-genai`, `pandas`, and `requests`. Do **not** create a virtual environment — it starts empty and hides packages the environment already provides, forcing a redundant install. Probe, and install only what is missing:

python3 -c "import vertexai, google.genai, pandas, requests" \
  || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'

The version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a redirect and silently writes an empty file instead of constraining the install.

Need `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`. Check env vars first; if missing, ask the user. Newer Gemini models often need `location="global"`.

Correct SDK entrypoints

import agentplatform
client = agentplatform.Client(project=PROJECT, location=LOCATION)

client.evals.run_inference(model=..., src=...)
client.evals.evaluate(dataset=..., metrics=...)
client.evals.generate_conversation_scenarios(...)

Two imports that look plausible and are not:

• `from agentplatform.types import evals` -- `ModuleNotFoundError`. `types` is

a module, not a package; use `from agentplatform import types`.

• `from vertexai.evaluation import PointwiseMetric, EvalTask` -- the

superseded SDK. Its classes take different arguments (`PointwiseMetric` has no `system_instruction`), so code written against it fails with `TypeError` rather than an import error. Use `agentplatform` throughout.

The Quality Flywheel

Five stages, run in order on the first pass, then loop 2 → 5 until quality targets are met.

Shortcuts that waste time

| Shortcut | Why it fails | | ------------------------------------ | ------------------------------------ | | "I'll tune the metric threshold down | Hides real failures. Fix the agent, | : so it passes." : not the bar. : | "This case is flaky, I'll skip it." | Flakiness reveals non-determinism in | : : the agent. Fix with `temperature=0` : : : or stricter instructions. : | "I just need to fix the eval | If expected outputs keep moving, the | : dataset, not the agent." : agent has a behavior problem. : | "I can tell from the trace it works | Self-grading doesn't generalize. | : — skip Stage 3." : Always run `evaluate()` and read : : : scores. : | "One iteration is enough." | Expect 5–10+ iterations. Stopping | : : early leaves regressions on other : : : metrics undetected. :

1. Prepare Data

Produce an `EvaluationDataset`. There are three input shapes, pick the one that matches the data the user already has:

• **`EvalCase` list (single-turn or multi-turn):**

    from agentplatform import types
    from google.genai import types as genai_types

    # prompt/reference/response values are Content, not str. UserContent and
    # ModelContent wrap a plain string and set the right role.
    dataset = types.EvaluationDataset