/agento11y-test-starter

name: agento11y-test-starter
description: >-
  Builds a starter offline test suite for an AI agent before it ships: test
  cases from its code, scoring advice, and a runner stub. Use when the agent has
  no real traffic yet. Not for tenant evaluators, rules, or guards (use
  agento11y-prod-setup).

agento11y test starter

Help a developer test an AI agent **before it ships** — while there is no real traffic yet. The hard part isn't running the test — it's having **cases to test against** and knowing **how to score them**, grounded in the agent's actual code.

> Scope: this is the **pre-production, offline** skill — it writes test cases and a local runner, > and never touches the tenant. Once the agent is deployed and has real traffic, setting up online > eval rules + guards on that traffic is a different skill, `agento11y-prod-setup`.

Always produce:

1. A ranked, justified **evaluator recommendation** for this agent. 2. A starter **suite YAML** the developer reviews and extends.

Then, depending on how runnable the agent is (Step 1):

3. For an easily-invoked agent, a **runner stub** (`run_experiment.py`) that wires the suite to the SDK with two holes to fill — and optionally run it (Step 6), only with permission. For an agent that needs a harness or full runtime, point to the existing eval infra instead of a runner that can't actually call it.

This skill is language-agnostic — the reading, recommending, and YAML it produces do not depend on the agent's language. What differs is how runnable the agent is: recommendations + YAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the agent has a clean function seam or needs a harness / full stack. For deeper run-side patterns (binding existing generations, cross-process verifiers) point to the per-language run skill (Python: `agento11y-experiments`).

> Note: `agento11y-experiments` currently ships in the grafana/agento11y repo > (`python/skills/agento11y-experiments/`), not in this gcx bundle yet — install it from there for > now. Consolidating it into the gcx bundle is pending.

Prerequisites

The generated runner imports the Agent Observability SDK. Install it in the agent's environment before running (Step 6):

• **Python:** `pip install agento11y python-dotenv` (the experiments API lives in

`agento11y.experiments`; the runner uses `python-dotenv` to load the agent's `.env`, and it is not a dependency of `agento11y`)

• **Go:** add `github.com/grafana/agento11y/go`

Only needed to *run* the suite (Step 6). Reading the agent, recommending evaluators, and writing the suite YAML (Steps 1–5) need nothing installed.

Rules

• Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No

control-plane writes. (Running an offline experiment only publishes that run's scores — it does not create tenant-level evaluators/rules/guards — but only do it via Step 6.)

• Do not rewrite the agent's prompt, optimize, or redeploy.
• Never run the experiment without asking first (Step 6). Never run against a target the

developer did not configure — use their `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN`; if the endpoint isn't set, ask for it, do not invent one.

• Never mint, generate, or store credentials. The developer owns the Grafana Cloud ingestion

token; read it from the environment (a gitignored `.env` or an exported env var they supply themselves) — **do not ask them to paste a secret token into the chat** (it is captured in the transcript), and do not create one.

• Never present the generated cases as validated. They are a draft to review and extend.
• **The `llm_judge` uses the LLM provider the agent already uses — don't add a new one.** If the

agent calls OpenAI, the judge calls OpenAI; if Anthropic, Anthropic. Do NOT default the judge to `litellm` (or any other provider SDK) when the app doesn't already depend on it — that adds a dependency and a second provider just for scoring. Reuse the agent's existing client/SDK.

• **The target is Grafana Cloud.** Publishing scores needs `AGENTO11Y_ENDPOINT` **and**

`AGENTO11Y_AUTH_TOKEN` (the ingestion key from the Connection page) — the SDK raises before making any request if the token is empty, so both are always required. Read the endpoint from an existing `AGENTO11Y_ENDPOINT` / `.env` / sibling app; never invent one or mint a token.

• If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.

Step 1 — Read the agent

Find and read these in the target repo, and record the file path and line range of each:

1. The agent entrypoint (where the model is invoked). 2. The system prompt / instructions. 3. The tool / function definitions the agent can call. 4. One or two real user requests and what a correct answer looks like. 5. **The LLM provider and the model** — two distinct things; read them off how the client is constructed and cite the line:

• **Provider** (who serves the LLM) → which SDK/client the judge reuses: e.g.

`from openai import OpenAI` → OpenAI; `from anthropic import Anthropic` → Anthropic.

• **Model** (the specific id, e.g. `gpt-4o-mini`, `claude-sonnet-5`) → the `MODEL_NAME` the runner uses.

State both explicitly, and have the judge use the **same provider** as the agent (never a different one). If you can't tell from the code, ask; don't default to a provider the app doesn't use.

• **Do NOT assume where the provider API key lives.** The app already loads it *somehow* — an env

var, a `.env` (`load_dotenv()`), a secret manager, an explicit `client(api_key=...)`. The runner should reuse the **same mechanism** (it already calls `load_dotenv()`, and provider SDKs read their standard env var themselves). Don't hardcode a key, and don't tell the developer to "set `OPENAI_API_KEY`" as if that's the only way — if the run can't find the key, ask them how their app loads it and mirror tha