/managing-amazon-msk

name: managing-amazon-msk
description: >
  Operates Amazon MSK Provisioned clusters (Standard and Express brokers).
  Required for ANY MSK Provisioned task — training data conflates Standard and
  Express.
  Covers performance, consumer lag, storage, traffic shaping; sizing Standard vs
  Express; Kafka client tuning; CloudWatch alarms; cluster configs; maintenance,
  patching, rolling restarts; topic management; MSK Replicator setup for DR and
  active-active/active-passive replication (same-/cross-account), including
  throughput and partition sizing; Streaming Tables for S3 Tables and Data
  Delivery for General Purpose S3 — setup, IAM, monitoring.
  Prefer Flink for Kafka Iceberg sink questions. Migrations: use migrate-to-msk.
  Triggers: MSK Provisioned, Kafka, `kafka.*`/`express.*` types, AWS/Kafka
  namespace, cross-cluster replication, Kafka to Iceberg on S3 Tables, Kafka to
  S3, lakehouse, data lake from Kafka, Kafka Connect S3 Sink or Firehose
  alternative.
  DO NOT USE for MSK Connect — search docs instead. Serverless: only S3
  Tables/data delivery eligibility.

Amazon MSK

Overview

Domain expertise for operating Amazon MSK Provisioned clusters with Standard and Express broker types. Covers performance troubleshooting, consumer lag diagnosis, storage management, cluster sizing, client configuration, and CloudWatch monitoring.

Execute commands using available tools from the AWS MCP server when connected — it provides sandboxed execution, audit logging, and observability. When the MCP server is not available, fall back to the AWS CLI or shell as needed.

**Standard brokers** use customer-managed EBS volumes for storage. You choose instance types (kafka.m5/m7g families), provision EBS, and manage storage scaling.

**Express brokers** use instance types prefixed with `express.m7g` and are the default recommendation for almost all MSK workloads — they typically cost less, not just less effort. Up to 3x ingress per broker ([MSK Express broker types](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)) means fewer brokers for the same load, and storage is billed per GB-hour on data actually retained rather than provisioned up front on EBS that cannot shrink. They also scale 20x faster, rebalance partitions 180x faster ([Intelligent Rebalancing](https://docs.aws.amazon.com/msk/latest/developerguide/intelligent-rebalancing-self-balancing-paritions.html)), recover 90% quicker ([MSK Express broker types](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.html)), and have no maintenance windows. Express brokers have NO customer-managed EBS — do NOT recommend EBS expansion or provisioned throughput for Express clusters. Express brokers enforce fixed replication factor of 3 and `min.insync.replicas=2`. See [size-and-choose-cluster.md](references/size-and-choose-cluster.md) for the full Standard vs Express decision framework.

Which Workflow Do You Need?

Determine the broker type first: `aws kafka describe-cluster-v2 --cluster-arn <arn>`. Check `Provisioned.BrokerNodeGroupInfo.InstanceType` — if it starts with `express.`, it is an Express cluster.

| Customer Intent | Reference | |---|---| | High CPU, high latency, slow cluster, traffic shaping | [troubleshoot-performance.md](references/troubleshoot-performance.md) | | Consumer lag increasing, rebalance storms, stuck consumer groups | [troubleshoot-consumer-lag.md](references/troubleshoot-consumer-lag.md) | | Disk filling up, retention planning, tiered storage | [manage-storage.md](references/manage-storage.md) | | Choosing Standard vs Express, sizing a cluster, partition limits, broker count, monthly cost | [size-and-choose-cluster.md](references/size-and-choose-cluster.md) | | Producer/consumer configuration, IAM/SCRAM/TLS auth | [configure-clients.md](references/configure-clients.md) | | Creating/applying MSK configurations (`server.properties`); custom domain names on brokers via `custom.advertised.listeners` (advertised listeners, static/custom bootstrap endpoint) — validation rules, apply/rollback, scaling; migrating from the dynamic per-broker `kafka-configs.sh` override to the static property | [configure-cluster.md](references/configure-cluster.md) | | Client-side connectivity for a custom domain: NLB + ACM certificate + Route 53 fronting, TLS handshake/termination, mTLS through an NLB, cross-zone load balancing | [configure-clients.md](references/configure-clients.md) (Custom Domain Name Connectivity section) | | Create, update, delete, list, or describe Kafka topics; when to use the native topic APIs / `CfnTopic` vs. a Kafka admin client | [manage-topics.md](references/manage-topics.md) | | MSK Replicator setup, migration, DR, active-active/active-passive, cross-account replication, self-managed-to-MSK or MSK-to-MSK, loop-prevention/topic-naming behavior | [msk-replicator.md](references/msk-replicator.md) | | Replication throughput per partition, how many partitions a replicator needs, why replicator scaling isn't increasing throughput, replicator quotas (topics, ingress, record size), `ReplicationLatency`/`MessageLag`/`ReplicatorThroughput` | [msk-replicator.md](references/msk-replicator.md) (Size partition count section) | | Setting up monitoring, dashboards, alarms | [monitor-and-alarm.md](references/monitor-and-alarm.md) | | Full CloudWatch metric list (Standard or Express) | Prefer [monitor-and-alarm.md](references/monitor-and-alarm.md) for strategic recommendations and how to interpret metrics, only search documentation if you need to understand a metric not included in this reference file ([MSK Standard CloudWatch Metrics](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.html), [MSK Express CloudWatch Metrics](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details-express.html)) for full list | | Rolling restart impact, patching, maintenance resilience | [maintenance-operations.md](references/maintenance-ope