real-time-streaming

When this skill is activated, always start your first response with the 🧢 emoji.

Real-Time Streaming

A practitioner's guide to building and operating real-time data pipelines. This skill covers the full stack of stream processing - from ingestion (Kafka producers, CDC with Debezium) through processing (Kafka Streams, Apache Flink) to materialization (sinks, materialized views, event-sourced stores). The focus is on production-grade patterns: exactly-once semantics, backpressure handling, state management, and failure recovery. Designed for engineers who understand distributed systems basics and need concrete guidance on building streaming pipelines that run reliably at scale.

When to use this skill

Trigger this skill when the user:

• Sets up or configures Kafka topics, producers, or consumers
• Writes a Flink job (DataStream or Table API, windowing, state)
• Implements change data capture (CDC) from a database to a streaming pipeline
• Designs a stream processing topology (joins, aggregations, windowing)
• Debugs consumer lag, rebalancing storms, or backpressure issues
• Implements exactly-once or at-least-once delivery guarantees
• Builds an event sourcing system with streaming infrastructure
• Needs to choose between Kafka Streams, Flink, or Spark Streaming

Do NOT trigger this skill for:

• General event-driven architecture decisions (use event-driven-architecture skill)
• Batch ETL pipelines with no real-time component (use a data-engineering skill)

Key principles

1. Treat streams as the source of truth - In a streaming architecture, the log

(Kafka topic) is the authoritative record. Databases, caches, and search indexes are derived views. Design from the stream outward, not from the database outward.

1. Partition for parallelism, key for correctness - Partitioning determines your

maximum parallelism. Key selection determines ordering guarantees. Choose partition keys based on your highest-volume access pattern. Events that must be processed in order must share a key (and therefore a partition).

1. Exactly-once is a system property, not a component property - No single

component delivers exactly-once alone. It requires idempotent producers, transactional writes, and consumer offset management working together end-to-end. Understand where your guarantees break down.

1. Backpressure is a feature, not a bug - When a consumer cannot keep up with a

producer, the system must signal this. Design pipelines with explicit backpressure handling rather than unbounded buffering. Flink handles this natively; Kafka consumers need careful tuning of max.poll.records and max.poll.interval.ms.

1. Late data is inevitable - Real-world events arrive out of order. Use watermarks

to define "how late is too late," allowed lateness windows to handle stragglers, and side outputs for events that arrive after the window closes.

Core concepts

The streaming stack has three layers. The transport layer (Kafka, Pulsar, Kinesis) provides durable, ordered, partitioned logs. The processing layer (Flink, Kafka Streams, Spark Structured Streaming) reads from the transport, applies transformations, and writes results. The materialization layer (databases, search indexes, caches) serves the processed data to applications.

Kafka's core model centers on topics divided into partitions. Producers write to partitions (by key hash or round-robin). Consumer groups read partitions in parallel - each partition is assigned to exactly one consumer in the group. Offsets track progress. Consumer group rebalancing redistributes partitions when consumers join or leave.

Flink's execution model is based on dataflow graphs. A job is a DAG of operators (sources, transformations, sinks). Flink manages state via checkpointing - periodic snapshots of operator state to durable storage. On failure, Flink restores from the last checkpoint and replays from the source offset, achieving exactly-once processing.

Change data capture (CDC) turns database changes into a stream of events. Debezium reads the database's transaction log (WAL for Postgres, binlog for MySQL) and publishes change events to Kafka. Each event contains before/after snapshots of the row, enabling downstream consumers to reconstruct the full change history.

Common tasks

Set up a Kafka topic with proper configuration

Choose partition count based on target throughput and consumer parallelism. Set replication factor to at least 3 for production.

kafka-topics.sh --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config cleanup.policy=delete \
  --config min.insync.replicas=2 \
  --bootstrap-server localhost:9092

Start with partitions = 2x your expected max consumer count. You can increase

partitions later but never decrease them. Changing partition count breaks key-based

ordering guarantees for existing data.

Write an idempotent Kafka producer (Java)

Enable idempotent production to prevent duplicates on retries.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", orderId, orderJson), (metadata, ex) -> {
    if (ex != null) log.error("Send failed for order {}", orderId, ex);
});

With enable.idempotence=true, the broker deduplicates retries using sequence

numbers. This requires acks=all and allows up to 5 in-flight requests while

maintaining ordering per partition.

Write a Flink windowed aggregation

Count events per key in tumbling 1-minute windows with late data handling.

DataStream<Event> events = env
    .addSource(new FlinkKafkaConsumer<>("clicks", new EventSchema(), kafkaProps))
    .assignTimestampsAndWatermarks(
        WatermarkStrategy.<Event>forBoundedOutOfOrderness(Duration.ofSeconds(10))
            .withTimestampAssigner((event, ts) -> event.getTimestamp()));

SingleOutputStreamOperator<WindowResult> result = events
    .keyBy(Event::getUserId)
    .window(TumblingEventTimeWindows.of(Time.minutes(1)))
    .allowedLateness(Time.minutes(5))
    .sideOutputLateData(lateOutputTag)
    .aggregate(new CountAggregator());

result.addSink(new JdbcSink<>(...));
result.getSideOutput(lateOutputTag).addSink(new LateDataSink<>());

Set forBoundedOutOfOrderness to the maximum expected event delay. Events arriving

within allowedLateness after the window fires trigger a re-computation. Events

arriving after that go to the side output.

Configure CDC with Debezium and Kafka Connect

Deploy a Debezium PostgreSQL connector to stream table changes.

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "db-primary",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${env:CDC_DB_PASSWORD}",
    "database.dbname": "commerce",
    "topic.prefix": "cdc",
    "table.include.list": "public.orders,public.order_items",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_orders",
    "publication.name": "dbz_orders_pub",
    "snapshot.mode": "initial",
    "transforms": "route",
    "transforms.route.type": "io.debezium.transforms.ByLogicalTableRouter",
    "transforms.route.topic.regex": "cdc\\.public\\.(.*)",
    "transforms.route.topic.replacement": "cdc.$1"
  }
}

Always