/setting-up-cloudwatch-observability

name: setting-up-cloudwatch-observability
description: >-
  Sets up CloudWatch observability for the first time - Omni (CloudWatch Application Observability) and
  classic CloudWatch. Omni: creating a Space or Domain; access grants (who has access, at what level),
  access profiles bounding alerts, integrations, agents; instrumenting an app or AI agent with the plain
  ADOT SDK so traces reach Omni (Python/Node/Java/.NET on EC2/ECS/EKS/Lambda), incl. no-image-rebuild,
  CoreCLR vars; ingesting Azure or Google Cloud telemetry via the CloudWatch agent (VM/AKS, GCE/GKE);
  connecting Slack or the AWS DevOps Agent to a Space; whether GitHub or a custom MCP tool server
  (HTTP/stdio) can connect. CloudWatch: onboarding a service to Application Signals: ADOT
  auto-instrumentation, amazon-cloudwatch-observability add-on, monitored service, ServiceEvents, CI/CD
  metadata, Terraform. For using what is set up - queries, dashboards, alarms, Omni alerts, X-Ray,
  synthetics, Dynamic Instrumentation, AI agent evaluation, detecting Omni vs classic - use
  aws-observability.

version: 4

Setting Up CloudWatch Observability

> **Scope:** First-time CloudWatch observability setup, for **both** products — a CloudWatch Application Observability (Omni) Space from creation through first traces flowing, **and** onboarding a service to classic CloudWatch **Application Signals**. For using what is already set up (queries, dashboards, alarms, Omni alerts, evaluations, live debugging), route to **aws-observability**.

Two products, two reference folders

This skill owns *setup* for two products that share the CloudWatch name but are separate services. The folder a reference lives in is the signal for which product it belongs to — the same convention `aws-observability` uses:

| Folder | Product | What setup means there | |---|---|---| | `references/cloudwatch-omni/` | CloudWatch Omni (Application / Agent Observability) | Domain → Space → grants → telemetry in → **plain ADOT SDK** instrumentation. No add-on, no CloudWatch Agent `application_signals` config, no port 4316. | | `references/cloudwatch/` | Classic CloudWatch | Onboarding a service to **Application Signals**: ADOT *auto*-instrumentation, the `amazon-cloudwatch-observability` EKS add-on, the CloudWatch Agent, monitored service, ServiceEvents, CI/CD git/deployment metadata, port 4316. |

The two instrumentation paths are **mutually exclusive on a given workload** — the Omni path explicitly forbids the Application Signals env vars and the add-on, and vice versa. Decide the product first (Routing Rules 1–4 below), then stay inside that folder. Enabling one does not replace the other; an account may run both, on different services.

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (`aws cloudwatchomni ...` on the Omni path; `aws eks`, `aws iam`, and `aws application-signals` on the Application Signals path).

Concepts — Omni (`references/cloudwatch-omni/`)

These are Omni's resources. **None of them exist on the Application Signals path** — that one has no Domain, Space, grant, or Dataset; its unit is a *monitored service*, and access is plain IAM. Do not ask an Application Signals onboarding customer for a `spaceId`.

| Term | What it is | |---|---| | **Domain** | The identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization. | | **Space** | A workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an account-scoped Identity Center Domain the Space must be in the Domain's own Region; under an org-scoped Identity Center Domain it must be in a Region the instance is replicated to. | | **Access grant** | Attaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data *through* a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM. | | **Access Profile** | A named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: `create-access-profile` takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it. | | **Dataset** | What queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups. |

Omni setup order from nothing: **Domain → Space → grants → telemetry in → instrumentation.** (The Application Signals equivalent is much shorter and has no prerequisite resources — add-on/agent, IAM, then the per-platform enablement change; see `references/cloudwatch/application-signals-onboarding.md`.) "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are **conditional**, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see `references/cloudwatch-omni/app-basics.md`.

This is a **routing skill**. Classify the user's setup request and delegate to the correct reference. Omni referenc