metrics-guide

Evaluation Metrics Reference

> File paths below reference the scaffolded layout (`tests/eval/eval_config.yaml` or `.json`). Adjust for your project structure if not using `google-agents-cli-scaffold`.

> Python projects scaffold the eval datasets in the `tests/eval/` directory; Go projects scaffold in the `eval/` directory. Examples below use the Python convention.

Managed (Built-in) Metrics Reference

Run `agents-cli eval metric list` for the live set. **Single-turn only** below means the metric 400s on a trace with 2+ turns (`Single-turn metric '<name>_v1' received agent_eval_data with N turns`). The single-turn adaptive-rubric metrics grade a case's own `rubric_groups` instead of generating their own when it supplies them (see *Managed Metric Parameters*).

Agent metrics (adaptive rubrics)

| Metric ID | Evaluates | Trace | |-----------|-----------|-------| | `multi_turn_task_success` | User goal/intent fulfillment across the conversation. Ignores supplied `rubric_groups`. | any | | `multi_turn_trajectory_quality` | Step sequencing, efficiency, error recovery. | any | | `multi_turn_tool_use_quality` | Technical and semantic correctness of tool calls. | any | | `final_response_quality` | Final response plus intermediate tool usage. | single-turn only | | `final_response_reference_free` | Final response quality with no reference answer. Needs `rubric_groups` on the case (500s without). | single-turn only | | `tool_use_quality` | Tool selection, parameter accuracy, step order. Needs `function_call` events in the trace. | single-turn only |

> `multi_turn_general_quality` and `multi_turn_text_quality` need a `conversation_history` field that `eval generate` does not produce, and 400 on agent traces. Use `multi_turn_task_success` or `multi_turn_trajectory_quality`.

General quality metrics (adaptive rubrics, single-turn only)

| Metric ID | Evaluates | |-----------|-----------| | `general_quality` | Overall quality with auto-generated criteria. Best starting point for non-agent eval. | | `text_quality` | Fluency, coherence, grammar. | | `instruction_following` | Adherence to the constraints in the prompt. |

Static rubric metrics (fixed criteria, single-turn only)

| Metric ID | Evaluates | |-----------|-----------| | `hallucination` | Segments the response into atomic claims and checks each against tool output. | | `final_response_match` | Judge-scored semantic match against a golden answer, not string equality. Needs `reference` on the case. | | `grounding` | Labels each sentence of the response supported or contradictory against context. Needs `context` (a string or `Content`) on the case. | | `safety` | Policy compliance (PII, hate speech, dangerous content, harassment, sexual). |

---

Custom Metrics

Custom metrics are declared in `eval_config.yaml` (or `.json`) under `custom_metrics`. See SKILL.md's *Evaluation Configuration Schema* section for how `metrics_to_run` selects from the pool. The schema below defines the per-entry fields.

Code-based metrics default to **local in-process execution** (no GCP project or region required); opt into the Vertex AI sandbox with `execution: "remote"`. The metric functions are written in Python, regardless of the project's language.

> **Scaffolded default metric.** The scaffolded `eval_config.yaml` ships `custom_response_quality` as a local LLM-judge in `tests/eval/response_quality.py` (referenced via `custom_function_file`, run in-process via `google-genai`). It grades on either backend — `genai.Client()` uses `GEMINI_API_KEY` (AI Studio) or ADC (Vertex) — and reads each case's `reference` (ground truth) when present. To grade with the managed Vertex eval service instead, replace it with a built-in metric or an `LLMMetric` (`prompt_template`).

Example

metrics_to_run:
  - multi_turn_trajectory_quality
  - project_response_rubric
  - agent_turn_count

custom_metrics:
  - name: project_response_rubric
    prompt_template: |
      Rate the agent's response 1-5 for helpfulness and accuracy.
      Prompt: {prompt}
      Final response: {response}
      Full trace (for tool-call and reasoning context): {agent_data}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model_sampling_count: 3

  - name: agent_turn_count
    custom_function: |
      def evaluate(instance):
          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {'score': len(turns)}

  - name: tool_call_count
    execution: remote
    custom_function: |
      def evaluate(instance):
          n = 0
          for turn in (instance.get("agent_data") or {}).get("turns", []):
              for event in turn.get("events", []):
                  for part in (event.get("content") or {}).get("parts", []):
                      if "function_call" in part:
                          n += 1
          return {'score': n}

Metrics receive the eval case's `{prompt}`, `{response}`, and `{agent_data}` (and `{reference}` / `{context}` when the case populates them) — see SKILL.md's *Evaluation Configuration Schema → Agent trace field model* for details.

Schema reference

Each entry in `custom_metrics` must conform to one of two Agent Platform evaluation metric schemas. `custom_function` or `custom_function_file` selects the code-based schema (in-process by default, `CodeExecutionMetric` with `execution: remote`); otherwise it's `LLMMetric`. An entry that carries neither, and whose `name` is a built-in metric, is a *managed metric parameterization* instead (see below).

Code Execution Metric (`CodeExecutionMetric`)

Evaluates responses using custom Python code.

| Field | Required | Description | |-------|----------|-------------| | `name` | yes | Unique identifier for the metric. | | `custom_function` | one of | Python source containing `def evaluate(instance):`. Receives an evaluation instance, returns a numeric score or a `{'score', 'explanation'}` dict. | | `custom_function_file` | one of | Path to