/dt-obs-tracing

name: dt-obs-tracing
description: >-
  Distributed traces, spans, service dependencies, and request flow analysis. Use when
  investigating span-level details, failures, performance bottlenecks, or trace correlation.
  Trigger: "trace analysis", "slow requests", "failed spans", "service dependencies",
  "distributed trace", "span details", "HTTP status codes in traces", "database query spans",
  "messaging spans", "gRPC calls", "Lambda cold starts", "trace ID lookup",
  "exception analysis", "correlate logs and traces", "request attributes".
  Do NOT use for explaining existing queries, product documentation or configuration questions,
  service-level RED metrics (use dt-obs-services), log searching (use dt-obs-logs),
  or problem analysis (use dt-obs-problems).
license: Apache-2.0

Application Tracing Skill

Overview

Distributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.

---

Use Cases

1. Investigate Slow Requests

• **Goal:** Find and diagnose requests exceeding a latency threshold
• **Trigger:** "slow requests", "high latency", "p99 response time", "find traces over 5 seconds"
• **Done:** List of slow traces with duration, endpoint, service, and trace IDs for drilldown

2. Analyze Request Failures

• **Goal:** Identify failed requests, failure reasons, and exception patterns
• **Trigger:** "failed spans", "HTTP 500 errors", "exception analysis", "failure rate by service"
• **Done:** Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces

3. Map Service Dependencies

• **Goal:** Understand service-to-service communication patterns and external API calls
• **Trigger:** "service dependencies", "what services does X call", "outgoing HTTP calls"
• **Done:** Dependency map showing call counts, latency, and error rates between services

---

Core Concepts

Understanding Traces and Spans

**Spans** represent logical units of work in distributed traces:

• HTTP requests, RPC calls, database operations
• Messaging system interactions
• Internal function invocations
• Custom instrumentation points

**Span kinds**:

• `span.kind: server` - Incoming call to a service
• `span.kind: client` - Outgoing call from a service
• `span.kind: consumer` - Incoming message consumption call to a service
• `span.kind: producer` - Outgoing message production call from a service
• `span.kind: internal` - Internal operation within a service

**Root spans**: A request root span (`request.is_root_span == true`) represents an incoming call to a service. Use this to analyze end-to-end request performance.

Key Trace Attributes

Essential attributes for trace analysis:

| Attribute | Description | |-----------|-------------| | `trace.id` | Unique trace identifier | | `span.id` | Unique span identifier | | `span.parent_id` | Parent span ID (null for root spans) | | `request.is_root_span` | Boolean, true for request entry points | | `request.is_failed` | Boolean, true if request failed | | `duration` | Span duration in nanoseconds | | `span.timing.cpu` | Overall CPU time of the span (stable) | | `span.timing.cpu_self` | CPU time excluding child spans (stable) | | `dt.smartscape.service` | Service Smartscape node ID | | `dt.service.name` | Dynatrace service name derived from service detection rules. It is equal to the Smartscape service node name. | | `endpoint.name` | Endpoint/route name |

Service Context

Spans reference services via Smartscape node IDs and the detected service name `dt.service.name` which is also present on every span.

fetch spans
| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }

**Node functions**:

• `getNodeName(dt.smartscape.service)` - Adds `dt.smartscape.service.name` field with the human-readable service name
• `getNodeField(dt.smartscape.service, "attribute_name")` - Access specific node attributes

**📖 Learn more**: See [Entity Lookups](references/entity-lookups.md) for advanced entity selectors, infrastructure correlation, and hardware analysis.

Sampling and Extrapolation

One span can represent multiple real operations due to:

• **Aggregation**: Multiple operations in one span (`aggregation.count`)
• **ATM (Adaptive Traffic Management)**: Head-based sampling by agent
• **ALR (Adaptive Load Reduction)**: Server-side sampling
• **Read Sampling**: Query-time sampling via `samplingRatio` parameter

**When to extrapolate**: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:

fetch spans
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| summarize operation_count = sum(multiplicity)

**📖 Learn more**: See [Sampling and Extrapolation](references/sampling-extrapolation.md) for detailed formulas and examples.

Common Query Patterns

Basic Span Access

Fetch spans and explore by type:

fetch spans | limit 1

Explore spans by function and type:

fetch spans
| summarize count(), by: { span.kind, code.namespace, code.function }

Request Root Filtering

List request root spans (incoming service calls):

fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100

Service Performance Summary

Analyze service performance with error rates:

fetch spans
| filter request.is_root_span == true
| summarize
    total_requests = count(),
    failed_requests = countIf(request.is_failed == true),