/dt-obs-services

name: dt-obs-services
description: >-
  Service performance monitoring with RED metrics (Rate, Errors, Duration) and runtime-specific
  telemetry for Java, .NET, Node.js, Python, PHP, and Go. Use when analyzing service health,
  SLA compliance, or runtime issues.
  Trigger: "service response time", "error rate", "throughput", "SLA compliance",
  "service mesh overhead", "JVM GC", "Java heap", "Node.js event loop", ".NET CLR",
  "Python threads", "PHP OPcache", "Go goroutines", "service performance",
  "p95 latency", "request failures", "database response time by name".
  Do NOT use for explaining existing queries, product documentation questions,
  infrastructure metrics (use dt-obs-hosts), log analysis (use dt-obs-logs),
  or distributed tracing workflows (use dt-obs-tracing).
license: Apache-2.0

Application Services Skill

Monitor application service performance, health, and runtime-specific metrics using DQL.

---

Core Capabilities

1. Service Performance (RED Metrics)

Monitor service **Rate, Errors, Duration** using metrics-based timeseries queries.

**Key Metrics:**

• `dt.service.request.response_time` - Response time (microseconds)
• `dt.service.request.count` - Request count
• `dt.service.request.failure_count` - Failed request count

**Common Use Cases:**

• Response time monitoring (avg, p50, p95, p99)
• Error rate tracking and spike detection
• Traffic analysis (throughput, peaks, growth)
• Performance degradation detection
• Multi-cluster comparison

**Quick Example:**

timeseries {
  p95 = percentile(dt.service.request.response_time, 95),
  total_requests = sum(dt.service.request.count),
  failures = sum(dt.service.request.failure_count)
}, by: {dt.service.name}
| fieldsAdd p95_ms = p95[] / 1000, error_rate_pct = (failures[] * 100.0) / total_requests[]

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

2. Advanced Service Analysis

Span-based queries for complex scenarios requiring flexible filtering and custom aggregations.

**Use Cases:**

• SLA compliance tracking with custom thresholds
• Service health scoring (multi-dimensional)
• Operation/endpoint-level performance analysis
• Custom error classification
• Failure pattern detection with error details

**Quick Example:**

fetch spans, from: now() - 1h | filter request.is_root_span == true
| fieldsAdd meets_sla = if(request.is_failed == false AND duration < 3s, 1, else: 0)
| summarize total = count(), sla_compliant = sum(meets_sla), by: {dt.service.name}
| fieldsAdd sla_compliance_pct = (sla_compliant * 100.0) / total

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

3. Service Messaging Metrics

Monitor message-based service communication (queues, topics).

**Key Metrics:**

• `dt.service.messaging.publish.count` - Messages sent to queues or topics
• `dt.service.messaging.receive.count` - Messages received from queues or topics
• `dt.service.messaging.process.count` - Messages successfully processed
• `dt.service.messaging.process.failure_count` - Messages that failed processing

**Use Cases:**

• Message throughput monitoring (publish/receive rates)
• Message processing failure tracking
• Queue/topic health analysis
• Consumer lag detection (publish vs receive rate comparison)

**Quick Example:**

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count),
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

4. Service Mesh Monitoring

Monitor service mesh ingress performance and overhead.

**Key Metrics:**

• `dt.service.request.service_mesh.response_time` - Mesh response time (microseconds)
• `dt.service.request.service_mesh.count` - Mesh request count
• `dt.service.request.service_mesh.failure_count` - Mesh failure count

**Use Cases:**

• Mesh vs direct performance comparison
• Mesh overhead calculation
• Mesh failure analysis
• gRPC traffic monitoring
• Multi-cluster mesh performance

**Quick Example:**

timeseries {
  direct_p95 = percentile(dt.service.request.response_time, 95),
  mesh_p95 = percentile(dt.service.request.service_mesh.response_time, 95)
}, by: {dt.service.name}
| fieldsAdd mesh_overhead_ms = (mesh_p95[] - direct_p95[]) / 1000

→ **For detailed queries:** See [references/service-metrics.md](references/service-metrics.md)

5. Runtime-Specific Monitoring

Technology-specific runtime performance and resource usage metrics.

**Java/JVM** - [references/java.md](references/java.md)

• Memory: heap, pools, metaspace
• GC: impact, suspension, frequency, pause time
• Threads: count monitoring, leak detection
• Classes: loading, unloading, growth

**Node.js** - [references/nodejs.md](references/nodejs.md)

• Event loop: utilization, active handles
• V8 heap: memory used, total
• GC: collection time, suspension
• Process: RSS memory

**.NET CLR** - [references/dotnet.md](references/dotnet.md)

• Memory: consumption by generation
• GC: collection count, suspension time
• Thread pool: threads, queued work
• JIT: compilation time

**Python** - [references/python.md](references/python.md)

• Threads: active thread count
• Heap: allocated blocks
• GC: collection by generation, pause time
• Objects: collected, uncollectable

**PHP** - [references/php.md](references/php.md)

• OPcache: hit ratio, memory, restarts
• GC: effectiveness, duration
• JIT: buffer usage
• Interned strings: usage, buffer

**Go** - [references/go.md](references/go.md)

• Goroutines: count, leak detection
• GC: suspension, collection time
• Memory: heap by state, committed
• Scheduler: worker threads, queue size
• CGo: call frequency

---

When to Use This Skill

✅ **Use for:**

• Monitoring service performance (response time, errors, traffic)
• C