/llm-obs

name: observability-llm-obs
description: >
  Answer questions about LLM and agentic-application behavior from data already ingested
  into Elastic: latency and error rate, token and cost utilization, response quality
  and guardrail events, and agentic call-chain orchestration. Use when the user asks
  about LLM monitoring, GenAI observability, token spend or AI cost, model latency,
  prompt or guardrail failures, or how an agent's tool-call chain executed.
compatibility: >
  Requires the `elastic` CLI (>= 0.2) with `es` and `kb` support, and an Elasticsearch
  deployment holding LLM telemetry from APM/OTLP traces or an Elastic LLM integration.
  Base floor is Elasticsearch 8.11+ or Serverless. The time-series path for integration
  metrics needs `TS` (Stack GA 9.4) and `TRANGE` (Stack GA 9.3); both are GA on Serverless.
  Below Stack 9.4 fall back to `FROM` with `BUCKET`. SLO and alerting lookups require
  Kibana. No Kibana UI is required.
metadata:
  author: elastic
  version: 0.3.1
  universal: true

LLM and Agentic Observability

Answer questions about monitoring LLMs and agentic components using **data actually ingested into Elastic** — nothing else. The four questions this skill answers are LLM performance, cost and token utilization, response quality, and call chaining or agentic workflow orchestration.

A given deployment typically uses **one or more** ingestion paths: APM/OTLP traces, and/or integration metrics and logs. Which one exists is a discovery result, not an assumption — never assume both are present. For ES|QL syntax, commands, and query patterns, use the **elasticsearch-esql** skill. For service-level latency and error triage that is not LLM-specific, use the **observability-sre-triage** skill.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Jobs to be done

• Discover which LLM ingestion path a deployment actually uses before querying anything
• Discover the real LLM field names in this deployment, because GenAI attribute naming is not consistent
• Report LLM latency, throughput, and error rate by model, service, or time
• Report token utilization, and report cost only when a cost field genuinely exists
• Investigate response quality: failures, timeouts, finish reasons, content filters, and guardrail events
• Reconstruct agentic call chains and find the bottleneck span in a multi-step workflow
• Correlate findings with SLOs and alerting rules defined on LLM-related data

Output discipline

Applies to every response produced under this skill.

• **Commit to the best-supported conclusion.** When the evidence points one way, say so. Do not downgrade confidence to

sound cautious — hedging on unambiguous evidence is a defect, not humility.

• **State confidence once**, in the conclusion. Do not restate it per bullet.
• **Do not speculate past the evidence.** If a cause was not observed, it does not go in the answer. Say what is unknown

and stop.

• **Report absence as absence.** Zero rows means the data is missing or not collected; it never means the underlying

condition is healthy. In particular, **if no cost field exists, say cost is not instrumented** — do not multiply token counts by a guessed per-token price and present the product as a cost figure.

• **Report the numbers actually found.** Quote the token counts, latencies, and error rates the query returned. Do not

round them into vague characterizations, and do not carry forward a number from an example in this skill.

• **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered, no

summarizing the summary.

• **End on the finding.** No trailing offers such as "want me to dig deeper?". Actionable follow-ups belong in a

recommendations list, phrased as recommendations, not as questions.

Where the data lives

| Ingestion path | Index patterns | What it can answer | | ------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | OTel / EDOT traces via OTLP | `traces-*.otel-*`, or the generic `traces-*` | Per-request latency, tokens, models, finish reasons, **and call chains** | | Elastic APM agent traces | `traces-apm*`, or the generic `traces-*` | Per-request latency and outcome; GenAI attributes if the SDK adds them | | APM/OTel metrics | `metrics-apm*`, `metrics-*.otel-*` | Agg