/agento11y-test-starter

name: agento11y-test-starter
description: >
  Use early in an AI-agent project — before ship, before real traffic — to build a starter
  test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools,
  task), writes a labeled draft suite of test cases (happy/edge/adversarial) grounded in real
  lines, and recommends how to score each case (the evaluators/judges the offline runner uses).
  Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub
  (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only
  against the endpoint the developer configured); for agents needing a harness or full runtime it
  points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators,
  rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic.
  Trigger on phrases like "how do I test my agent before shipping", "write test cases for my
  agent", "set up tests for my agent", "check my agent before prod", "I have no traffic yet, how
  do I evaluate it", "test my agent offline".

agento11y test starter

Help a developer test an AI agent **before it ships** — while there is no real traffic yet. The hard part isn't running the test — it's having **cases to test against** and knowing **how to score them**, grounded in the agent's actual code.

> Scope: this is the **pre-production, offline** skill — it writes test cases and a local runner, > and never touches the tenant. Once the agent is deployed and has real traffic, setting up online > eval rules + guards on that traffic is a different skill, `agento11y-prod-setup`.

Always produce:

1. A ranked, justified **evaluator recommendation** for this agent. 2. A starter **suite YAML** the developer reviews and extends.

Then, depending on how runnable the agent is (Step 1):

3. For an easily-invoked agent, a **runner stub** (`run_experiment.py`) that wires the suite to the SDK with two holes to fill — and optionally run it (Step 6), only with permission. For an agent that needs a harness or full runtime, point to the existing eval infra instead of a runner that can't actually call it.

This skill is language-agnostic — the reading, recommending, and YAML it produces do not depend on the agent's language. What differs is how runnable the agent is: recommendations + YAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the agent has a clean function seam or needs a harness / full stack. For deeper run-side patterns (binding existing generations, cross-process verifiers) point to the per-language run skill (Python: `agento11y-experiments`).

> Note: `agento11y-experiments` currently ships in the grafana/agento11y repo > (`python/skills/agento11y-experiments/`), not in this gcx bundle yet — install it from there for > now. Consolidating it into the gcx bundle is pending.

Prerequisites

The generated runner imports the Agent Observability SDK. Install it in the agent's environment before running (Step 6):

• **Python:** `pip install agento11y python-dotenv` (the experiments API lives in

`agento11y.experiments`; the runner uses `python-dotenv` to load the agent's `.env`, and it is not a dependency of `agento11y`)

• **Go:** add `github.com/grafana/agento11y/go`

Only needed to *run* the suite (Step 6). Reading the agent, recommending evaluators, and writing the suite YAML (Steps 1–5) need nothing installed.

Rules

• Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No

control-plane writes. (Running an offline experiment only publishes that run's scores — it does not create tenant-level evaluators/rules/guards — but only do it via Step 6.)

• Do not rewrite the agent's prompt, optimize, or redeploy.
• Never run the experiment without asking first (Step 6). Never run against a target the

developer did not configure — use their `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN`; if the endpoint isn't set, ask for it, do not invent one.

• Never mint, generate, or store credentials. The developer owns the Grafana Cloud ingestion

token; read it from the environment (a gitignored `.env` or an exported env var they supply themselves) — **do not ask them to paste a secret token into the chat** (it is captured in the transcript), and do not create one.

• Never present the generated cases as validated. They are a draft to review and extend.
• **The `llm_judge` uses the LLM provider the agent already uses — don't add a new one.** If the

agent calls OpenAI, the judge calls OpenAI; if Anthropic, Anthropic. Do NOT default the judge to `litellm` (or any other provider SDK) when the app doesn't already depend on it — that adds a dependency and a second provider just for scoring. Reuse the agent's existing client/SDK.

• **The target is Grafana Cloud.** Publishing scores needs `AGENTO11Y_ENDPOINT` **and**

`AGENTO11Y_AUTH_TOKEN` (the ingestion key from the Connection page) — the SDK raises before making any request if the token is empty, so both are always required. Read the endpoint from an existing `AGENTO11Y_ENDPOINT` / `.env` / sibling app; never invent one or mint a token.

• If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.

Step 1 — Read the agent

Find and read these in the target repo, and record the file path and line range of each:

1. The agent entrypoint (where the model is invoked). 2. The system prompt / instructions. 3. The tool / function definitions the agent can call. 4. One or two real user requests and what a correct answer looks like. 5. **The LLM provider and the model** — two distinct things; read them off how the client is constructed and cite the line:

• **Provider** (who serves the LLM) → which SDK/client the judge reuses: e.g.

`from openai import OpenAI` → OpenAI; `from anthropic import Anthropic` → Anthropic. -