/dt-obs-problems

name: dt-obs-problems
description: >-
  DAVIS problem analysis including root cause identification, impact assessment,
  and correlation with other telemetry. Use when querying or investigating detected problems.
  Trigger: "active problems", "root cause analysis", "problem impact", "affected users",
  "list problems", "P-12345 details", "recurring problems", "problem history",
  "problem trending", "blast radius", "which entity caused the problem",
  "problems affecting Kubernetes", "problems by service".
  Do NOT use for explaining existing queries, product documentation questions, generic
  log searching, distributed tracing, or host-level resource monitoring.
license: Apache-2.0

Problem Analysis Skill

Analyze Dynatrace AI-detected problems including root cause identification, impact assessment, and correlation with logs and metrics.

---

Use Cases

1. Active Problem Triage

• **Goal:** List and prioritize currently active problems
• **Trigger:** "active problems", "what problems are open", "current issues", "availability issues"
• **Done:** Prioritized list of active problems with category, user impact, and display IDs

2. Root Cause Investigation

• **Goal:** Identify the root cause entity for a specific problem
• **Trigger:** "root cause of P-12345", "what caused this problem", "which entity is the root cause"
• **Done:** Root cause entity identified with affected entity list and blast radius

3. Problem Trending

• **Goal:** Analyze problem patterns over time to identify recurring issues
• **Trigger:** "recurring problems", "problem history", "problem trends last 30 days"
• **Done:** Trend data showing problem frequency, recurring root causes, and resolution times

---

Overview

Dynatrace automatically detects anomalies, performance degradations, and failures across your environment, creating **problems** that aggregate related alert, warning and info-level events and provide root cause and impact insights.

What are Problems?

Problems are automatically detected, software and infrastructure health and resilience issues that:

• **Automatically correlate** related alert, warning, and info-level events across services, infrastructure, frontend applications, and user sessions
• **Identify root causes** using causal analysis of Smartscape dependencies
• **Assess business impact** by tracking affected users and services
• **Reduce alert noise** by grouping related symptoms into single problems that share the same root cause and impact
• **Track problem lifecycle** from early detection through resolution

Event Kinds

The `event.kind` field (stable, permission) identifies the high-level event type:

| `event.kind` value | Description | |---|---| | `DAVIS_EVENT` | Davis-detected infrastructure/application events | | `BIZ_EVENT` | Business events (ingested via API or captured from spans) | | `RUM_EVENT` | Real User Monitoring events | | `AUDIT_EVENT` | Administrative/security audit events |

`event.provider` (stable, permission) identifies the event source.

Problem Categories

Common `event.category` values:

| Category | Description | Example | |----------|-------------|---------| | **AVAILABILITY** | Infrastructure or service unavailable | Web service returns no data, synthetic test actively fails, database connection lost | | **ERROR** | Increased error rates beyond baseline | API error rate jumped from 0.1% to 15% | | **SLOWDOWN** | Performance degradation | Response time increased from 200ms to 5000ms | | **RESOURCE** | Resource saturation | Container memory at 95%, causing OOM kills | | **CUSTOM** | Custom anomaly detections | Business KPI (orders/minute) dropped below threshold |

Problem Lifecycle

Detection → ACTIVE → Under Investigation → CLOSED

• **ACTIVE**: Currently occurring issues requiring attention
• **CLOSED**: Resolved issues used for historical analysis

Essential Fields

Common Field Name Mistakes

| ❌ WRONG | ✅ CORRECT | Description | |---------|-----------|-------------| | `title` | `event.name` | Problem title/description | | `status` | `event.status` | Problem lifecycle status | | `severity` | `event.category` | Problem type/category | | `start` | `event.start` | Problem start time |

Correct Status Values

// ✅ CORRECT: Use these status values
fetch dt.davis.problems
| filter event.status == "ACTIVE"   // Currently occurring problems
//     or event.status == "CLOSED"  // Resolved problems
// ❌ INCORRECT: event.status == "OPEN" does not exist!
| limit 1

Key Fields Reference

fetch dt.davis.problems, from:now() - 1h
| filter not(dt.davis.is_duplicate)
| fields
    event.start,                          // Problem start timestamp
    event.end,                            // Problem end timestamp (if closed)
    display_id,                           // Human-readable problem ID (P-XXXXX)
    event.name,                           // Problem title
    event.description,                    // Detailed description
    event.category,                       // Problem type
    event.status,                         // ACTIVE or CLOSED
    dt.smartscape_source.id,              // The smartscape ID for the affected resource
    dt.davis.affected_users_count,        // Number of affected users
    affected_entity_ids = smartscape.affected_entities[][id],  // Array of affected entity IDs
    dt.smartscape.service,                // Affected services (may be array)
    dt.davis.root_cause_entity,           // Entity identified as root cause
    root_cause_entity_id,                 // Root cause entity ID
    root_cause_entity_name,               // Human-readable root cause name
    dt.davis.is_duplicate,                // Whether duplicate detection
    dt.davis.is_rootcause                 // Root cause vs. symptom
| limit 10

Standard Query Pattern

Always start problem queries with this foundation:

fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_dupli