/streaming-data

name: streaming-data
description: Build event streaming and real-time data pipelines with Kafka, Pulsar, Redpanda, Flink, and Spark. Covers producer/consumer patterns, stream processing, event sourcing, and CDC across TypeScript, Python, Go, and Java. When building real-time systems, microservices communication, or data integration pipelines.

Streaming Data Processing

Build production-ready event streaming systems and real-time data pipelines using modern message brokers and stream processors.

When to Use This Skill

Use this skill when:

• Building event-driven architectures and microservices communication
• Processing real-time analytics, monitoring, or alerting systems
• Implementing data integration pipelines (CDC, ETL/ELT)
• Creating log or metrics aggregation systems
• Developing IoT platforms or high-frequency trading systems

Core Concepts

Message Brokers vs Stream Processors

**Message Brokers** (Kafka, Pulsar, Redpanda):

• Store and distribute event streams
• Provide durability, replay capability, partitioning
• Handle producer/consumer coordination

**Stream Processors** (Flink, Spark, Kafka Streams):

• Transform and aggregate streaming data
• Provide windowing, joins, stateful operations
• Execute complex event processing (CEP)

Delivery Guarantees

**At-Most-Once**:

• Messages may be lost, no duplicates
• Lowest overhead
• Use for: Metrics, logs where loss is acceptable

**At-Least-Once**:

• Messages never lost, may have duplicates
• Moderate overhead, requires idempotent consumers
• Use for: Most applications (default choice)

**Exactly-Once**:

• Messages never lost or duplicated
• Highest overhead, requires transactional processing
• Use for: Financial transactions, critical state updates

Quick Start Guide

Step 1: Choose a Message Broker

See references/broker-selection.md for detailed comparison.

**Quick decision**:

• **Apache Kafka**: Mature ecosystem, enterprise features, event sourcing
• **Redpanda**: Low latency, Kafka-compatible, simpler operations (no ZooKeeper)
• **Apache Pulsar**: Multi-tenancy, geo-replication, tiered storage
• **RabbitMQ**: Traditional message queues, RPC patterns

Step 2: Choose a Stream Processor (if needed)

See references/processor-selection.md for detailed comparison.

**Quick decision**:

• **Apache Flink**: Millisecond latency, real-time analytics, CEP
• **Apache Spark**: Batch + stream hybrid, ML integration, analytics
• **Kafka Streams**: Embedded in microservices, no separate cluster
• **ksqlDB**: SQL interface for stream processing

Step 3: Implement Producer/Consumer Patterns

Choose language-specific guide:

• TypeScript/Node.js: references/typescript-patterns.md (KafkaJS)
• Python: references/python-patterns.md (confluent-kafka-python)
• Go: references/go-patterns.md (kafka-go)
• Java/Scala: references/java-patterns.md (Apache Kafka Java Client)

Common Patterns

Basic Producer Pattern

Send events to a topic with error handling:

1. Create producer with broker addresses
2. Configure delivery guarantees (acks, retries, idempotence)
3. Send messages with key (for partitioning) and value
4. Handle delivery callbacks or errors
5. Flush and close producer on shutdown

Basic Consumer Pattern

Process events from topics with offset management:

1. Create consumer with broker addresses and group ID
2. Subscribe to topics
3. Poll for messages
4. Process each message
5. Commit offsets (auto or manual)
6. Handle errors (retry, DLQ, skip)
7. Close consumer gracefully

Error Handling Strategy

For production systems, implement:

• **Dead Letter Queue (DLQ)**: Send failed messages to separate topic
• **Retry Logic**: Configurable retry attempts with backoff
• **Graceful Shutdown**: Finish processing, commit offsets, close connections
• **Monitoring**: Track consumer lag, error rates, throughput

Decision Frameworks

Framework: Message Broker Selection

START: What are requirements?

1. Need Kafka API compatibility?
   YES → Kafka or Redpanda
   NO → Continue

2. Is multi-tenancy critical?
   YES → Apache Pulsar
   NO → Continue

3. Operational simplicity priority?
   YES → Redpanda (single binary, no ZooKeeper)
   NO → Continue

4. Mature ecosystem needed?
   YES → Apache Kafka
   NO → Redpanda (better performance)

5. Task queues (not event streams)?
   YES → RabbitMQ or message-queues skill
   NO → Kafka/Redpanda/Pulsar

Framework: Stream Processor Selection

START: What is latency requirement?

1. Millisecond-level latency needed?
   YES → Apache Flink
   NO → Continue

2. Batch + stream in same pipeline?
   YES → Apache Spark Streaming
   NO → Continue

3. Embedded in microservice?
   YES → Kafka Streams
   NO → Continue

4. SQL interface for analysts?
   YES → ksqlDB
   NO → Flink or Spark

5. Python primary language?
   YES → Spark (PySpark) or Faust
   NO → Flink (Java/Scala)

Framework: Language Selection

**TypeScript/Node.js**:

• API gateways, web services, real-time dashboards
• KafkaJS library (827 code snippets, high reputation)

**Python**:

• Data science, ML pipelines, analytics
• confluent-kafka-python (192 snippets, score 68.8)

**Go**:

• High-performance microservices, infrastructure tools
• kafka-go (42 snippets, idiomatic Go)

**Java/Scala**:

• Enterprise applications, Kafka Streams, Flink, Spark
• Apache Kafka Java Client (683 snippets, score 76.9)

Advanced Patterns

Event Sourcing

Store state changes as immutable events. See references/event-sourcing.md for:

• Event store design patterns
• Event schema evolution
• Snapshot strategies
• Temporal queries and audit trails

Change Data Capture (CDC)

Capture database changes as events. See references/cdc-patterns.md for:

• Debezium integration (MySQL, PostgreSQL, MongoDB)
• Real-time data synchronization
• Microservices data integration patterns

Exactly-Once Processing

Implement transactional guarantees. See references/exactly-once.md for:

• Idempotent pro