/agento11y-prod-setup

name: agento11y-prod-setup
description: >
  Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana
  Agent Observability, grounded in the agent's own code and its real ingested traffic.
  The judgment layer on top of the `agento11y` skill: it reads the agent's source
  (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks
  what evaluators/rules/guards already exist, then recommends only what's missing —
  online eval rules (score live conversations for regressions) and guards (warn-first
  request-path policies that redact / tool-filter and may later be promoted to deny).
  It drafts reviewable YAML and, only with explicit confirmation, applies via
  `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but
  never blocks). It DOES create stack-level objects —
  that is the point — but every write is confirmed. It never rewrites or redeploys the
  agent. Trigger on phrases like "set up production evaluation", "my agent is in prod
  what should I evaluate", "catch quality regressions", "add guardrails to my agent",
  "redact PII from my agent", "block dangerous tools", "set up online evals and guards".
allowed-tools: Bash, Read, Write, Edit

Agent Observability — production evals & guards setup

The production counterpart to `agento11y-test-starter` (which runs pre-ship, on code alone, producing an offline test suite). This skill runs **after** ship, when the agent has real traffic, and sets up the two production surfaces the starter deliberately leaves out:

• **Online eval rules** — evaluators that score ingested live conversations, so regressions

surface without hand-reviewing every conversation.

• **Guards** (hook-rules) — policies on the request path that `warn` (and can later be promoted to

`deny`) in real time. A guard decides via one of three shapes: `evaluator_ids` (an evaluator judges), `redact` (regex redaction), or `tool_filter` (block tool calls). See Step 4.

What this skill does that `agento11y` doesn't

The sibling `agento11y` skill is the **mechanics** layer: exact CLI flags, evaluator/rule YAML shapes, create-or-update semantics, the online-eval setup steps. It assumes you already know *what* to create.

This skill is the **judgment** layer. It answers *which* rules and guards this specific agent needs, by grounding in two evidence sources a generic checklist can't use:

1. **The agent's code** — its system prompt, tools, and how it handles user data. Half the value is here; read and cite it (`file:line`). 2. **The agent's real traffic** — because it's deployed, you can see what it actually does in prod, not just what the code says it might.

Two gaps this skill fills beyond `agento11y`:

• **Recommendation from evidence** — `agento11y` starts once you know what to create; this decides.
• **Guards** — `agento11y` documents evaluators and rules but not guards (hook-rules), even though

`gcx agento11y guards` exists. This skill carries the guard shapes (`evaluator_ids` / `redact` / `tool_filter`, plus `action_on_fail`) itself.

For any mechanical detail — exact flags, evaluator/rule YAML fields, the setup flow — defer to the `agento11y` skill and to `gcx agento11y <sub> --help` rather than restating it here.

Rules

• Every connection to the stack goes through **`gcx agento11y`** — never raw HTTP, never a hand-held

token. `gcx` owns Cloud auth (via `gcx login`). Prerequisite: `gcx` installed and authenticated; if it isn't, say so and stop.

• **Confirm the target stack before any WRITE (Step 0 + Step 5).** Reads run freely once you've

shown the context; writes (upsert evaluators, create/update rules and guards) need an explicit yes on the target stack. `gcx` may be pointed at the wrong stack, and this skill creates stack-level objects.

• **Check before recommending.** Always list what already exists first

(`gcx agento11y evaluators list`, `rules list`, `guards list`) and never recommend a duplicate. Compare by **semantic equivalence**, not just id/name — see Step 2.

• This skill **does** create stack-level objects — that is its job, the one thing that separates

it from `agento11y-test-starter`. But every creation is **explicit and confirmed**: show the exact YAML, get a yes, then create it with the matching `gcx agento11y` command. A yes for one object is not a yes for the next.

• New guards are always drafted **`action_on_fail: "warn"`** — even hard-policy ones. That is what

makes a new guard safe: a `warn` guard only records the outcome, it never blocks a request, so it is harmless even while it is live. Never draft a first-time `deny` guard. The developer switches to `deny` themselves, later, after watching it in warn mode (Step 6). Draft `enabled: false` if you can, but don't build extra steps around it — the server may store the guard `enabled: true` regardless (see Step 5), and that is fine, because `warn` carries the safety, not `enabled`.

• New online rules start with a **conservative `sample_rate`** (e.g. `0.1`), not `1.0` — an

`llm_judge` over 100% of traffic costs real money.

• Prefer **starting from an evaluator template** (`gcx agento11y templates list`, then

`gcx agento11y templates get`) over authoring a new evaluator. Only write a fresh one when nothing fits.

• Do not rewrite the agent's prompt, optimize, or redeploy. This skill configures observation and

guardrails around the agent, not the agent itself.

• If a required input is missing (the agent's name as Agent Observability sees it, or `gcx` auth),

ask — don't guess.

Step 0 — Confirm the target stack

Before reading traffic or writing anything, show the developer where `gcx` is pointed. The active context may not be the stack they think:

gcx config current-context        # the active context name
gcx config view                   # its server URL, org-id, auth method

Display the resolved **context name, server URL, and org-id**. Two t