/agento11y-prod-setup

name: agento11y-prod-setup
description: >-
  Sets up production evaluation for a deployed AI agent in Agent Observability:
  online eval rules and guards (redact PII, block tools), based on its code and
  live traffic. Applies via gcx after confirmation. Not for pre-ship tests (use
  agento11y-test-starter).
allowed-tools: Bash, Read, Write, Edit

Agent Observability — production evals & guards setup

The production counterpart to `agento11y-test-starter` (which runs pre-ship, on code alone, producing an offline test suite). This skill runs **after** ship, when the agent has real traffic, and sets up the two production surfaces the starter deliberately leaves out:

• **Online eval rules** — evaluators that score ingested live conversations, so regressions

surface without hand-reviewing every conversation.

• **Guards** (hook-rules) — policies on the request path that `warn` (and can later be promoted to

`deny`) in real time. A guard decides via one of three shapes: `evaluator_ids` (an evaluator judges), `redact` (regex redaction), or `tool_filter` (block tool calls). See Step 4.

What this skill does that `agento11y` doesn't

The sibling `agento11y` skill is the **mechanics** layer: exact CLI flags, evaluator/rule YAML shapes, create-or-update semantics, the online-eval setup steps. It assumes you already know *what* to create.

This skill is the **judgment** layer. It answers *which* rules and guards this specific agent needs, by grounding in two evidence sources a generic checklist can't use:

1. **The agent's code** — its system prompt, tools, and how it handles user data. Half the value is here; read and cite it (`file:line`). 2. **The agent's real traffic** — because it's deployed, you can see what it actually does in prod, not just what the code says it might.

Two gaps this skill fills beyond `agento11y`:

• **Recommendation from evidence** — `agento11y` starts once you know what to create; this decides.
• **Guards** — `agento11y` documents evaluators and rules but not guards (hook-rules), even though

`gcx agento11y guards` exists. This skill carries the guard shapes (`evaluator_ids` / `redact` / `tool_filter`, plus `action_on_fail`) itself.

For any mechanical detail — exact flags, evaluator/rule YAML fields, the setup flow — defer to the `agento11y` skill and to `gcx agento11y <sub> --help` rather than restating it here.

Rules

• Every connection to the stack goes through **`gcx agento11y`** — never raw HTTP, never a hand-held

token. `gcx` owns Cloud auth (via `gcx login`). Prerequisite: `gcx` installed and authenticated; if it isn't, say so and stop.

• **Confirm the target stack before any WRITE (Step 0 + Step 5).** Reads run freely once you've

shown the context; writes (upsert evaluators, create/update rules and guards) need an explicit yes on the target stack. `gcx` may be pointed at the wrong stack, and this skill creates stack-level objects.

• **Check before recommending.** Always list what already exists first

(`gcx agento11y evaluators list`, `rules list`, `guards list`) and never recommend a duplicate. Compare by **semantic equivalence**, not just id/name — see Step 2.

• This skill **does** create stack-level objects — that is its job, the one thing that separates

it from `agento11y-test-starter`. But every creation is **explicit and confirmed**: show the exact YAML, get a yes, then create it with the matching `gcx agento11y` command. A yes for one object is not a yes for the next.

• New guards are always drafted **`action_on_fail: "warn"`** — even hard-policy ones. That is what

makes a new guard safe: a `warn` guard only records the outcome, it never blocks a request, so it is harmless even while it is live. Never draft a first-time `deny` guard. The developer switches to `deny` themselves, later, after watching it in warn mode (Step 6). Draft `enabled: false` if you can, but don't build extra steps around it — the server may store the guard `enabled: true` regardless (see Step 5), and that is fine, because `warn` carries the safety, not `enabled`.

• New online rules start with a **conservative `sample_rate`** (e.g. `0.1`), not `1.0` — an

`llm_judge` over 100% of traffic costs real money.

• Prefer **starting from an evaluator template** (`gcx agento11y templates list`, then

`gcx agento11y templates get`) over authoring a new evaluator. Only write a fresh one when nothing fits.

• Do not rewrite the agent's prompt, optimize, or redeploy. This skill configures observation and

guardrails around the agent, not the agent itself.

• If a required input is missing (the agent's name as Agent Observability sees it, or `gcx` auth),

ask — don't guess.

Step 0 — Confirm the target stack

Before reading traffic or writing anything, show the developer where `gcx` is pointed. The active context may not be the stack they think:

gcx config current-context        # the active context name
gcx config view                   # its server URL, org-id, auth method

Display the resolved **context name, server URL, and org-id**. Two thresholds:

• **Before reads** (traffic sampling, inventory): show the resolved context so the developer sees

where the discovery ran. A wrong stack here wastes effort but doesn't change anything.

• **Before any write** (Step 5): require an **explicit yes** that this is the intended production

stack. Writes are what create stack-level objects, so this confirmation is the hard gate.

If it's wrong, stop — the developer switches with `gcx config use-context <name>`, or you pass `--context <name>` on every `gcx` call. (Watch for `localhost` / dev-looking servers — a strong sign the active context is not their prod stack.)

Step 1 — Read the code and sample real traffic

Two evidence sources. Do both; every later recommendation cites one of them.

**Code** (as `agento11y-test-starter` Step 1). Find and record file:line for: the entrypoint, the system prompt, the tool/function defin