/aws-observability

name: aws-observability
description: >-
  Investigates root causes on the Omni context graph (service dependencies) and Intelligence insights;
  builds, configures, debugs, tunes AWS observability - operator-symptom questions, detecting Omni vs
  classic CloudWatch. CloudWatch on a reporting service: fleet health views, Log Insights,
  metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries,
  CloudTrail, Dynamic Instrumentation (live breakpoints), and Application Signals service map. CloudWatch
  Omni on an existing Space: SQL over logs/traces, PromQL, Omni dashboards, alerts and Slack
  notifications, programmatic/IaC access (API/SDK/CLI/CFN), driving Omni from a coding agent or skills, AI
  agent quality from traces - on-demand and online evaluation of live traffic, incl. setting it up,
  readback, evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT
  instrumentation for Application Signals or Omni - use setting-up-cloudwatch-observability. Not for app
  logging or threat detection.

metadata:
  version: "12"

AWS Observability

Overview

Domain expertise for AWS observability across metrics, logs, and traces, for **two products** that share the CloudWatch name but are separate services with separate control planes, data models, and APIs:

| | CloudWatch | CloudWatch Omni | |---|---|---| | **What it is** | Log groups, metric namespaces, alarms, Log Insights, X-Ray, Application Signals | Application Observability / Agent Observability. A **Space** per account per Region is the access boundary over the account's CloudWatch **Dataset** (OpenTelemetry logs, traces, and metrics); the Dataset is a CloudWatch resource the Space reads, not something the Space contains | | **Control plane** | `aws cloudwatch`, `aws logs`, `aws xray`, `aws application-signals` | `aws cloudwatchomni` (endpoint prefix `cloudwatch-omni`, signing name `cloudwatch`) | | **Query** | Log Insights query language; GetMetricData | SQL over `logs.default` / `traces.default`; PromQL over metrics; named views | | **Notify** | Alarms (metric, composite, anomaly) | **Alerts** (SQL/PromQL rule, contributors, OK/WARNING/CRITICAL/NODATA) | | **Topology** | Application Signals service map | **Context graph** (GetContextGraph) | | **Access** | IAM only | Domain → Space → **access grants** and **access profiles** (set up in `setting-up-cloudwatch-observability`) | | **Only here** | Dynamic Instrumentation, Synthetics canaries, CloudTrail auditing, EMF | Agent-quality evaluation, views, context graph | | **First-time setup** | Application Signals onboarding — ADOT auto-instrumentation, the `amazon-cloudwatch-observability` add-on, ServiceEvents, CI/CD metadata | Domain → Space → grants → telemetry in → plain-ADOT instrumentation | | **References** | `references/cloudwatch/` | `references/cloudwatch-omni/` |

**This skill covers *using* both products, never setting either one up.** Both first-time setup paths above live in **`setting-up-cloudwatch-observability`** (same two-folder split); route there rather than describing an onboarding procedure from memory.

Enabling Omni does not replace CloudWatch; log groups, metrics, and alarms keep working, and most customers use both.

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) — enables running CLI commands, querying CloudWatch, and validating configurations directly. All guidance also works with standard AWS CLI access.

**Note:** Reference files contain specific runtime versions, quota values, and feature matrices that may change. When precision matters (e.g. deploying to production, choosing a runtime, or checking a quota), confirm values against current AWS documentation rather than relying solely on the values in these files.

Scope guard — is this an observability request?

This skill owns the scope decision for every request routed to it, and this section is its sole home.

**If a request is NOT about AWS observability with CloudWatch or CloudWatch Omni (weather, trivia, general chit-chat, non-observability coding), decline it in one sentence: state plainly that it is out of scope, never fabricate an answer, and never claim a false capability limitation (no "no internet access", no "no weather data") — the reason is simply that it is out of scope. Then redirect by naming what this skill does cover: querying CloudWatch and CloudWatch Omni logs, traces, and metrics, building dashboards, configuring alerts, investigating a service, or agent evaluation.**

Do **not** attempt the off-topic task, and do **not** call a tool or run a query in pursuit of it. Keep it brief: no lecture, no long refusal.

Step 0 — CloudWatch or CloudWatch Omni?

Decide this before routing. The natural wording ("set up an alert for high latency", "build a dashboard", "query my logs") does not say which product the customer means.

1. **The customer names the product** — "Omni", "Application Observability", "Agent Observability", a Space, Domain, Dataset, access grant, `spaceId`, `cloudwatch-omni`, Omni SQL, PromQL, views, context graph, evaluators, agent evaluation / scoring traces / online or continuous evaluation / `gen_ai.evaluation`, and OTel span vocabulary (`traces.default`, `logs.default`, spans, `durationNano`, `status.code`, resource attributes, `service.name`) → **Omni**. "Log Insights", "log group", "metric namespace", "CloudWatch alarm", "metric alarm", "composite alarm", "anomaly alarm", "X-Ray", "Application Signals", "canary", "CloudTrail", "Dynamic Instrumentation" → **CloudWatch**. A "log group" named inside an agent-evaluation request is the online-evaluation data source to verify, not a CloudWatch signal. A bare "alarm" (or "alert") with no other product signal is ambiguous — fall through to rule 3 and probe: a Space → Omni alert ([alerts.md](references/cloudwatch-omni/alerts.md)); no Space → CloudWatch alarm ([cloudwatch/alar