/dt-obs-genai

name: dt-obs-genai
description: >-
  Analyze & debug GenAI/LLM apps: token cost & caching by prompt, model &
  provider; latency/errors; agent & tool loops/failures; conversations;
  guardrails; evaluations; OpenTelemetry/dt-evals setup.
license: Apache-2.0

AI Observability (GenAI) Skill

Analyze AI Observability signals from customer GenAI applications using DQL — golden signals, LLM signals, token and cost analytics (with usage attribution and prompt-caching economics), agent signals (including loop/runaway detection and Smartscape topology), conversation/session-level analytics, guardrails, and evaluation quality.

When to Use

Use this skill for observability questions about customer GenAI applications — anything reading OpenTelemetry GenAI spans (`gen_ai.*`) or LLM evaluation bizevents. Example triggers:

• "LLM latency", "error rate by model"
• "token usage by model", "token throughput / TPM", "am I hitting rate limits", "provider throttling or 429s"
• "cost by model and provider", "who is driving token spend", "do I have prompt caching"
• "cost per conversation", "most expensive sessions"
• "failing agent tool calls", "find runaway agents"
• "responses truncated or blocked", "finish reasons"
• "failed evaluations", "quality scores"

When NOT to Use

Davis CoPilot/MCP telemetry (dt-platform), generic service metrics (dt-obs-services), logs (dt-obs-logs), or non-GenAI distributed tracing (dt-obs-tracing).

Example Questions

When suggesting follow-up questions (e.g., "give me one example question per topic"), use these canonical, ready-to-ask phrasings — one per capability. Keep suggestions single-clause and avoid the literal phrase "content filter" (use "blocked or safety-filtered" instead); overly long, multi-clause questions can be rejected.

• **Traffic, errors & latency:** What is my LLM error rate and p95 latency by model in the last 24 hours?
• **Token usage & cost:** Break down token usage and cost by model and provider in the last 24 hours.
• **Agent & tool activity:** Show me failing agent tool calls in the last 24 hours.
• **Conversation analytics:** What are my top 10 most expensive conversations by total token usage in the last 24 hours?
• **Guardrails:** How many responses were blocked or truncated in the last 7 days, and which model was affected most?
• **Evaluation quality:** Show me failed evaluations from the last 24 hours with the judge's explanation and the question-answer pair.

---

Core Capabilities

Golden Signals

The four classic observability signals — traffic, errors, latency, and saturation — apply directly to GenAI applications. Traffic is request throughput over time; errors are spans where `span.status_code == "error"`; latency is the `duration` field (a Grail duration value — divide by the `1ms` literal, `duration / 1ms`, for a numeric millisecond value); saturation is proxied by total token throughput per minute (input + output tokens combined).

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc

→ **Full traffic, latency, and saturation queries:** See [references/golden-signals.md](references/golden-signals.md)

LLM Signals

LLM signals describe which model and provider served each request, what operation type was invoked (`chat`, `execute_tool`, `invoke_agent`, `create_agent`), and how tokens were consumed. Use these to benchmark provider latency, compare model performance, and understand the token distribution across model-provider combinations.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc

→ **Slowest models, token usage by model:** See [references/llm-signals.md](references/llm-signals.md)

Cost and Tokens

Token consumption is the primary cost driver. Dynatrace stores `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` on every span — there is no stored cost field; estimated cost must be derived by multiplying token sums by the per-model price you supply. Use these queries to identify the highest-spend model-provider combinations and detect token-burn spikes.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc

→ **Token spikes, cost estimation, most expensive prompts, usage attribution, prompt-caching economics:** See [references/cost-and-tokens.md](references/cost-and-tokens.md)

Agent Signals

GenAI agents emit spans for each tool invocation (`execute_tool`), agent step (`invoke_agent`), and agent creation (`create_agent`). Use agent signals to identify which tools are called most often and which agents are failing. For *structural* questions — which agents, models, and providers exist and how they connect — query the GenAI Smartscape entities (`GENAI_AGENT`, `GENAI_MODEL`, `GENAI_PROVIDER`, `GENAI_SERVICE`) instead of scanning spans; this is a feature-flag-gated preview.

fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc

→ **Tool usage, failing agents, agent step latency, loop/runaway detection, Smartscape topology:** See [references/agent-signals.md](references/agent-signals.md)

Conversation Analytics

Per-span and per-trace signals measure one