dataset_schema

Evaluation Dataset Schema

Canonical formats for evaluation datasets in the Agent Platform Evaluation SDK. The summary below covers the type tree as of the version this skill targets — for the live, authoritative definitions see the public SDK source: [`types/evals.py`](https://github.com/googleapis/python-aiplatform/blob/main/agentplatform/_genai/types/evals.py) and [`types/common.py`](https://github.com/googleapis/python-aiplatform/blob/main/agentplatform/_genai/types/common.py).

Core Types

EvaluationDataset
└── eval_cases: list[EvalCase]       # List of evaluation cases

EvalCase
├── prompt: Content                          # Single-turn: the user query
├── responses: list[ResponseCandidate]       # Single-turn: model response(s); list to support multi-candidate eval
├── reference: ResponseCandidate             # Ground truth, needed by `final_response_match`
├── context: str | Content                   # Source text, needed by `grounding`
├── agent_data: AgentData                    # Multi-turn: full conversation trajectory
├── rubric_groups: dict[str, RubricGroup]    # Per-case rubrics; graded by managed rubric metrics
└── (extra fields allowed)                   # Custom fields for custom metrics

ResponseCandidate
└── response: Content                # The actual Content (role + parts)

AgentData
├── agents: dict[str, AgentConfig]   # Agent definitions
└── turns: list[ConversationTurn]    # Ordered conversation turns

ConversationTurn
├── turn_index: int                  # 0-based turn number
└── events: list[AgentEvent]         # Events within this turn

AgentEvent
├── author: str                      # "user", agent_id, or "tool"
└── content: Content                 # Content with role and parts

> **Note on `responses` and `reference`.** Both wrap a `Content` inside a `ResponseCandidate` object. So a single-turn case writes `"responses": [{"response": {"role": "model", "parts": [...]}}]` and `"reference": {"response": {"role": "model", "parts": [...]}}` — NOT a bare `Content`. `prompt` and `agent_data.turns[].events[].content` are bare `Content` (not wrapped).

Single-Turn Dataset

For simple prompt-response evaluation (e.g., QA, summarization).

{
  "eval_cases": [
    {
      "eval_case_id": "capital_of_france",
      "prompt": {
        "role": "user",
        "parts": [{"text": "What is the capital of France?"}]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [{"text": "The capital of France is Paris."}]
          }
        }
      ],
      "reference": {
        "response": {
          "role": "model",
          "parts": [{"text": "Paris"}]
        }
      }
    },
    {
      "eval_case_id": "summarize_article",
      "prompt": {
        "role": "user",
        "parts": [{"text": "Summarize this article: ..."}]
      },
      "responses": [
        {
          "response": {
            "role": "model",
            "parts": [{"text": "The article discusses..."}]
          }
        }
      ]
    }
  ]
}

Required fields by metric type

| Metric category | Required fields | |---|---| | Predefined (single-turn) | `prompt`, `responses` | | Computation-based | `responses`, `reference` | | Translation | `prompt` (source), `responses`, `reference` | | Custom LLM/code | Fields referenced in your template/function |

Multi-Turn / Multi-Agent Dataset

For evaluating multi-turn agent conversations, including systems with multiple collaborating agents and tool calls. The `agents` map declares all participating agents; `turns` is the chronological conversation, where each `event` author is `"user"`, an agent ID from the `agents` map, or `"tool"`.

{
  "eval_cases": [
    {
      "eval_case_id": "flight_booking_via_specialist",
      "agent_data": {
        "agents": {
          "router": {
            "agent_id": "router",
            "agent_type": "RouterAgent",
            "instruction": "Route requests to the appropriate specialist."
          },
          "flight_bot": {
            "agent_id": "flight_bot",
            "agent_type": "SpecialistAgent",
            "instruction": "Search and book flights.",
            "tools": [{
              "function_declarations": [{
                "name": "search_flights",
                "description": "Search flights by destination",
                "parameters": {
                  "type": "OBJECT",
                  "properties": {
                    "destination": {"type": "STRING"}
                  }
                }
              }]
            }]
          }
        },
        "turns": [
          {
            "turn_index": 0,
            "events": [
              {
                "author": "user",
                "content": {
                  "parts": [{"text": "Book a flight to NYC"}]
                }
              },
              {
                "author": "router",
                "content": {
                  "parts": [{"text": "Routing to flight_bot."}]
                }
              }
            ]
          },
          {
            "turn_index": 1,
            "events": [
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{
                    "function_call": {
                      "name": "search_flights",
                      "args": {"destination": "NYC"}
                    }
                  }]
                }
              },
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{
                    "function_response": {
                      "name": "search_flights",
                      "response": {"flights": [{"id": "AA123", "price": 320}]}
                    }
                  }]
                }
              },
              {
                "author": "flight_bot",
                "content": {
                  "parts": [{"text": "Found AA123 to NYC for $320.