/aws-cloudwatch-investigation

name: aws-cloudwatch-investigation
description: >
  Reusable investigation patterns for AWS CloudWatch: Logs Insights query templates,
  alarm-to-deployment correlation, blast-radius narrowing decision tree, and
  PromQL-style metric query patterns for structured incident triage.

AWS CloudWatch Investigation Skill

Reusable patterns for investigating production incidents using CloudWatch Logs, Metrics, and Alarms. These patterns are designed to be composed together during incident triage.

---

Pattern 1: Logs Insights Query Templates

Error Spike Detection

Find the top errors in a time window, grouped by error type:

fields @timestamp, @message, @logStream
| filter @message like /(?i)(error|exception|fatal|critical)/
| stats count(*) as errorCount by bin(5m), @logStream
| sort errorCount desc
| limit 20

P99 Latency Breakdown by Operation

Identify which operations are driving latency spikes:

fields @timestamp, @duration, operation
| filter ispresent(@duration)
| stats avg(@duration) as avgMs,
        pct(@duration, 50) as p50Ms,
        pct(@duration, 95) as p95Ms,
        pct(@duration, 99) as p99Ms,
        count(*) as invocations
  by operation
| sort p99Ms desc
| limit 15

Lambda Cold Start Detection

Quantify cold start impact during an incident:

fields @timestamp, @duration, @initDuration, @memorySize, @maxMemoryUsed
| filter ispresent(@initDuration)
| stats count(*) as coldStarts,
        avg(@initDuration) as avgInitMs,
        max(@initDuration) as maxInitMs,
        avg(@duration) as avgDurationMs
  by bin(5m)
| sort @timestamp desc

Out-of-Memory (OOM) Detection

Find Lambda functions or containers killed by memory pressure:

fields @timestamp, @message, @logStream, @memorySize, @maxMemoryUsed
| filter @message like /Runtime exited|out of memory|OOMKilled|Cannot allocate memory|MemoryError/
| stats count(*) as oomEvents by @logStream, bin(10m)
| sort oomEvents desc
| limit 10

For memory utilization trending before OOM:

fields @timestamp, @maxMemoryUsed, @memorySize
| filter ispresent(@maxMemoryUsed)
| stats max(@maxMemoryUsed / @memorySize * 100) as peakMemPct,
        avg(@maxMemoryUsed / @memorySize * 100) as avgMemPct
  by bin(5m)
| sort @timestamp desc

Timeout Detection

Find invocations that hit the configured timeout:

fields @timestamp, @duration, @logStream, @requestId
| filter @message like /Task timed out/ or @duration > 28000
| stats count(*) as timeouts by @logStream, bin(5m)
| sort timeouts desc

---

Pattern 2: Alarm History to Deploy-Event Correlation

Process

1. **Get alarm transition time** — note the exact timestamp when the alarm entered ALARM state. 2. **Query CloudTrail** for deployment-related events in a window of [alarm_time - 30min, alarm_time]:

# CloudTrail Lake query for deployment events
SELECT eventTime, eventName, userIdentity.arn, requestParameters
FROM <event-data-store-id>
WHERE eventTime > '<alarm_time_minus_30m>'
  AND eventTime < '<alarm_time>'
  AND eventName IN (
    'UpdateFunctionCode', 'UpdateFunctionConfiguration',
    'UpdateService', 'CreateDeployment', 'RegisterTaskDefinition',
    'CreateChangeSet', 'ExecuteChangeSet',
    'StartPipelineExecution', 'PutImage'
  )
ORDER BY eventTime DESC

3. **Correlation criteria** — a deploy is "correlated" if:

• It targets the same service/resource as the alarm
• It completed within 15 minutes before the alarm transition
• The deployer identity matches a CI/CD role (not a human applying a hotfix)

4. **Strengthening the correlation:**

• Check if the same alarm was healthy in the previous deployment cycle
• Verify no other environmental changes (scaling events, config changes) in the same window
• Look for canary/synthetic monitor failures that started at the same time

Output Format

Deploy Correlation:
  Event: UpdateFunctionCode
  Time: 2024-03-15T14:23:07Z (12 min before alarm)
  Actor: arn:aws:sts::123456789012:assumed-role/github-actions-deploy/session
  Resource: arn:aws:lambda:us-east-1:123456789012:function:payment-processor
  Correlation: STRONG — same resource, CI/CD actor, alarm was OK prior cycle

---

Pattern 3: Narrow the Blast Radius Decision Tree

Use this tree to systematically scope an incident from broadest to most specific:

START
  |
  v
[1] ACCOUNT — Which account(s) show the alarm?
  |  - Check: Are alarms firing in multiple accounts?
  |  - If yes → suspect shared service (SSO, networking, shared deployment pipeline)
  |  - If no → proceed to Region
  v
[2] REGION — Which region(s) are affected?
  |  - Check: Same alarm in other regions?
  |  - If multi-region → suspect global service (IAM, Route53, S3 global)
  |  - If single-region → proceed to Service
  v
[3] SERVICE — Which service namespace shows degradation?
  |  - Check CloudWatch namespace: AWS/Lambda, AWS/ECS, AWS/ApiGateway, etc.
  |  - If multiple services → suspect shared dependency (VPC, NAT, DNS, IAM)
  |  - If single service → proceed to Operation
  v
[4] OPERATION — Which API action or function is failing?
  |  - For Lambda: which function name?
  |  - For ECS: which service/task definition?
  |  - For API GW: which stage/resource/method?
  |  - If all operations → suspect service-level issue (throttling, quota)
  |  - If specific operation → proceed to Resource
  v
[5] RESOURCE — Which specific resource instance?
     - Function ARN, Task ID, DB instance identifier
     - This is your investigation target
     - Proceed to log and trace analysis scoped to this resource

Shared Dependency Investigation

When blast radius spans multiple services, investigate in this order:

1. **VPC/Networking** — NAT Gateway ErrorPortAllocation, packet drops, DNS resolution failures 2. **IAM/STS** — ThrottlingException on AssumeRole, token vending latency 3. **Downstream dependency** — shared database, cache, or external API 4. **Deployment pipeline** — simultaneous deploys