/google-agents-cli-eval

name: google-agents-cli-eval
description: >
  This skill should be used when the user wants to "run an evaluation",
  "evaluate my ADK agent", "write an eval dataset", "analyze eval failures",
  "compare eval results", "optimize agent", or needs guidance on the Agent Platform
  eval methodology and the Quality Flywheel.
  Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes.
  Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment
  (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
metadata:
  author: Google
  license: Apache-2.0
  version: 1.3.1
  requires:
    bins:
      - agents-cli
    install: "uv tool install google-agents-cli"

Agent Evaluation Guide

> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https://docs.astral.sh/uv/getting-started/installation/index.md) first if needed.

> **Scaffolded project?** If you used `/google-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests/eval/datasets/`, and `tests/eval/eval_config.yaml`. Start with executing `eval run` and iterate from there.

Reference Files

| File | Contents | |------|----------| | `references/dataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn / multi-turn / multi-agent, common mistakes | | `references/metrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config | | `references/user-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics | | `references/builtin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility | | `references/multimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |

---

The Quality Flywheel

Improving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.

1. Prepare Data

**Default:** Use or edit the scaffolded `tests/eval/datasets/basic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.

**Opt-in:** `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See *Eval Commands* and `references/user-simulation.md`.

2. Run Inference

`agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts/traces/`. Run this when you wrote the dataset by hand in Stage 1 (default path). **Skip this stage if you used `eval dataset synthesize`** — that command already produced traces.

3. Grade Traces (always run)

`agents-cli eval grade` — scores the traces and writes `results_<ts>.{json,html}` to `artifacts/grade_results/`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces.

> **Shortcut:** `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts/traces/` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file.

4. Analyze Failures

**Default:** Open the latest `artifacts/grade_results/results_<ts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table.

**Opt-in:** `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See *Eval Commands*.

5. Optimize & Code Fix

**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.

**Opt-in:** `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.

> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.

Running the loop

Iterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare <prev_results>.json <new_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes — this is normal. Only after a case passes should you expand coverage with more eval cases.

When doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.

Shortcuts That Waste Time

Recognize these rationalizations and push back — they always cost more time than they save:

| Shortcut | Why it fails | |----------|-------------| | "I'll tune the eval thresholds down to make it pass" | Lowering thresholds hides real failures. If the agent can't meet the bar, fix the agent — don't move the bar. | | "This eval case is flaky, I'll skip it" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. | | "I just need to fix the eval dataset, not the agent" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |

Choosing