/agento11y-instrument

name: agento11y-instrument
description: >-
  Instruments a developer's own LLM app or repairs missing generations and empty
  Performance charts in Grafana Agent Observability. Applies confirmed changes
  and verifies generation ingest, traces, and metrics. Excludes coding-agent
  telemetry plugins. For offline tests use agento11y-test-starter; for eval rules
  and guards use agento11y-prod-setup.
allowed-tools: Bash, Read, Grep, Glob, Edit, Write, WebFetch

Agent Observability — instrument an LLM app

Help a developer wire **their own** LLM app or agent to Grafana Agent Observability, from zero or from a broken/partial state, and **keep going until data actually lands in the stack**. The value this skill adds over the static instrumentation prompt is two things a prompt can't do:

1. A mechanical **gap checklist** run against the real code — headed by the one failure that is completely silent (missing OTel providers → every metric lost, no error). 2. A **verification loop** through `gcx`: after each change, run the app and check the `gcx agento11y` agents / conversations / generations commands to confirm generations arrived. Diagnose the next gap from what's missing, not from guesswork.

The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps) lives in agento11y's `llms.txt` "Path B". Fetch it rather than restating it here; this file holds the flow and the decision logic. A minimal fallback lives in [references/instrumentation.md](references/instrumentation.md) for when the fetch is unavailable.

Rules

• **Reference, don't restate.** Fetch SDK detail from

`https://raw.githubusercontent.com/grafana/agento11y/main/llms.txt` (Path B). Only inline decision logic here. If the fetch fails, fall back to [references/instrumentation.md](references/instrumentation.md).

• **Never invent an endpoint or a token.** Read them from the environment (`AGENTO11Y_ENDPOINT`,

`AGENTO11Y_PROTOCOL`, `AGENTO11Y_AUTH_MODE`, `AGENTO11Y_AUTH_TENANT_ID`, `AGENTO11Y_AUTH_TOKEN`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`) or ask the developer. Never fabricate a URL or mint a token.

• **Target is Grafana Cloud.** The developer supplies the endpoint + token (Step 0), and the gcx

verification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the endpoint or token — read them from the environment or ask.

• **Write `AGENTO11Y_*` env vars, never `SIGIL_*`.** `SIGIL_*` is a deprecated legacy fallback. Do

this **even if sibling apps or existing `.env` files in the repo use `SIGIL_*`** — matching a stale local convention perpetuates it. If the app already reads `SIGIL_*`, add the `AGENTO11Y_*` names (the SDK still honors both) and note the old ones are deprecated. Do not "match the siblings."

• **Gate every code WRITE on explicit confirmation.** Report first (Step 4), apply only after the

developer says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside the loop; editing files does not.

• **Keep diffs small; do not change app behavior.** Instrumentation is additive. No refactors, no

prompt rewrites, no dependency upgrades beyond the SDK/adapter packages actually needed.

• **Never change the model, provider, or the app's LLM config — not even with permission, not even

"just to run the verify loop."** Instrument whatever model the app already uses. This is absolute: changing the model is out of scope for instrumentation, full stop. If a run fails because a provider API key is missing, the only allowed responses are: (a) ask the developer to provide the key for the model the app *already* uses, or (b) skip the live run and report the wiring as verified-by-construction, telling the developer to run it themselves. Do **not** offer to switch the provider, do **not** ask "which provider should I use?", and do **not** add a new provider dependency (e.g. `langchain-anthropic`) to make the run succeed. If the developer separately says they *want* a different model, that is an app change they own — tell them to make it and re-invoke this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what the app does and what gets observed, which defeats the point. **The provider API key (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, …) is the app's own concern, not the instrumentation's** — it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the two 401s: a 401 on **generation ingest** is observability auth and *is* yours to fix (usually a missing `AGENTO11Y_PROTOCOL`/`AGENTO11Y_AUTH_MODE`); an auth error from the **model provider** is not — surface it and let the developer handle their own key.

• **Do not assume language symmetry.** Verify the provider wrapper / framework adapter actually

exists for the app's language before recommending it (Python has the most adapters, JS fewer, Go only google-adk, Java/.NET core + providers + google-adk). If it doesn't exist, hand-instrument with the core SDK. Prefer, in order: **provider wrapper → framework adapter → hand-instrumentation**.

• **The loop is bounded.** At most ~3–4 instrument→verify iterations. If data still isn't landing,

stop and report what's checked and what remains — don't loop forever.

• **Field-name traps:** `cache_write_input_tokens`, NOT `cache_creation_input_tokens`. `agent_version`

maps to the `gen_ai.agent.version` label and is required for per-version Performance charts. **`MessageRole` (Python SDK) has only `USER`, `ASSISTANT`, `TOOL` — there is no `SYSTEM` (or `DEVELOPER`) member**; `MessageRole.SYSTEM` raises `AttributeError`. Fold the system prompt into the `USER` message (or a `text_part`), and pr