Apache Kafka: The Definitive Guide to Distributed Event Streaming

Apache Kafka: The Definitive Guide to Distributed Event Streaming

Apache Kafka is a distributed event streaming platform capable of handling trillions of events per day. Originally developed at LinkedIn and open-sourced in 2011, Kafka has become the de facto standard for real-time data pipelines, event sourcing, and stream processing. Unlike traditional message queues that delete messages after consumption, Kafka retains messages for a configurable period, enabling multiple consumers to read the same data independently and even replay historical events.

Kafka Architecture

Brokers

A Kafka cluster consists of one or more servers called brokers. Each broker stores data and serves client requests. Brokers are stateless regarding consumers — they do not track which messages have been consumed. Instead, consumers manage their own position (offset) in the log.

Topics and Partitions

Data in Kafka is organized into topics — named feeds of messages. Each topic is divided into one or more partitions. Partitions are the fundamental unit of parallelism and scaling in Kafka.

# Topic: "orders" with 6 partitions across 3 brokers
#
# Broker 1:  Partition 0  |  Partition 3
# Broker 2:  Partition 1  |  Partition 4
# Broker 3:  Partition 2  |  Partition 5
#
# Each partition is an ordered, immutable sequence of messages:
# Partition 0: [msg0, msg1, msg2, msg3, msg4, ...]
#                                              ^ newest
# Partition 1: [msg0, msg1, msg2, ...]
#
# Messages within a partition are ordered; across partitions, no ordering guarantee

Producers

Producers publish messages to topics. Each message is appended to a specific partition based on a partitioning strategy:

# Python Kafka producer example
from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['kafka-1:9092', 'kafka-2:9092', 'kafka-3:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    key_serializer=lambda k: k.encode('utf-8') if k else None,
    acks='all',                    # Wait for all replicas to acknowledge
    retries=3,                     # Retry on transient failures
    linger_ms=10,                  # Batch messages for 10ms
    batch_size=16384               # Max batch size in bytes
)

# Publish with key (ensures same key always goes to same partition)
producer.send(
    'orders',
    key='customer-1001',           # Partition key
    value={
        'order_id': 'ORD-5001',
        'customer_id': 'customer-1001',
        'items': [{'sku': 'WIDGET-1', 'qty': 2}],
        'total': 59.98,
        'timestamp': '2024-01-15T10:30:00Z'
    }
)

# Messages with key "customer-1001" always go to the same partition
# This guarantees ordering for all of customer-1001's orders
producer.flush()

Consumer Groups

Consumers read from topics as part of a consumer group. Each partition is consumed by exactly one consumer within a group, enabling parallel processing. Different consumer groups read the same data independently.

# Consumer Group: "order-processing" (3 consumers, 6 partitions)
#
# Consumer A: reads Partition 0, Partition 1
# Consumer B: reads Partition 2, Partition 3
# Consumer C: reads Partition 4, Partition 5
#
# Consumer Group: "analytics" (2 consumers, 6 partitions)
#
# Consumer X: reads Partition 0, 1, 2
# Consumer Y: reads Partition 3, 4, 5
#
# Both groups read ALL messages independently

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['kafka-1:9092'],
    group_id='order-processing',
    auto_offset_reset='earliest',    # Start from beginning if no offset
    enable_auto_commit=False,        # Manual offset management
    value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)

for message in consumer:
    order = message.value
    partition = message.partition
    offset = message.offset
    
    try:
        process_order(order)
        consumer.commit()            # Commit offset after successful processing
        print(f"Processed: partition={partition}, offset={offset}")
    except Exception as e:
        print(f"Error processing order: {e}")
        # Do not commit — message will be reprocessed

Offset Management

Offsets are the mechanism Kafka uses to track consumer progress. Each message in a partition has a sequential offset number. Consumers commit their current offset to tell Kafka which messages they have processed.

Retention and Storage

Unlike traditional queues, Kafka retains messages even after they are consumed. Retention is configured per topic:

# Kafka topic configuration
# Retain messages for 7 days (default)
retention.ms=604800000

# Retain up to 1 TB per partition
retention.bytes=1073741824

# Compact topic — keep only latest value per key (great for changelog)
cleanup.policy=compact

# Both time-based and size-based retention
cleanup.policy=delete
retention.ms=604800000
retention.bytes=107374182400

# Create topic with specific configuration
kafka-topics.sh --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2

Replication and Fault Tolerance

Each partition is replicated across multiple brokers. One replica is the leader (handles reads and writes) and others are followers (replicate from the leader). If the leader fails, a follower is promoted.

# Replication factor of 3: each partition has 3 copies
#
# Partition 0:
#   Leader:   Broker 1
#   Follower: Broker 2 (in-sync)
#   Follower: Broker 3 (in-sync)
#
# ISR (In-Sync Replicas): replicas that are caught up with the leader
# min.insync.replicas=2: at least 2 replicas must acknowledge a write
# Combined with acks=all: guarantees no data loss even if one broker fails

# Key broker configuration
num.partitions=6                  # Default partitions per topic
default.replication.factor=3      # Default replication
min.insync.replicas=2             # Minimum ISR for writes to succeed
unclean.leader.election.enable=false  # Prevent data loss on failover

Common Use Cases

• Event Sourcing: Store all state changes as events in Kafka. Reconstruct state by replaying events. Used for event-driven architectures and audit trails.
• Real-Time Analytics: Feed clickstream, transaction, and sensor data into Kafka for stream processing with tools like Kafka Streams or Apache Flink.
• Change Data Capture (CDC): Stream database changes into Kafka using Debezium or similar connectors. Downstream services react to database changes in real-time.
• Log Aggregation: Collect logs from hundreds of services into Kafka, then fan out to Elasticsearch (search), S3 (archival), and monitoring systems.
• Microservice Integration: Services communicate through Kafka topics instead of direct API calls, achieving full decoupling.

Kafka vs Traditional Message Queues

Frequently Asked Questions

How many partitions should I create for a topic?

Start with the number of consumers you expect in your largest consumer group. If you need 12 consumers processing in parallel, create at least 12 partitions. Having more partitions than consumers is fine (each consumer gets multiple partitions). More partitions increase parallelism but also increase broker overhead, rebalancing time, and end-to-end latency. A common starting point is 6-12 partitions per topic.

How does Kafka achieve such high throughput?

Kafka uses sequential disk I/O (which is faster than random I/O), zero-copy data transfer (sendfile syscall), batching at the producer and consumer, and efficient compression. Messages are written to an append-only log — no random seeks. Combined with the partition model for parallelism, this allows a single Kafka cluster to handle millions of messages per second.

What is the difference between Kafka and Kafka Streams?

Kafka is the messaging platform (brokers, topics, producers, consumers). Kafka Streams is a client library for stream processing — it reads from Kafka topics, processes data (filters, transforms, joins, aggregates), and writes results back to Kafka topics. It is a lightweight alternative to Apache Flink or Spark Streaming that runs as a regular Java application without a separate cluster.

How do I handle exactly-once semantics in Kafka?

Kafka supports exactly-once semantics (EOS) since version 0.11 through idempotent producers and transactional writes. Set enable.idempotence=true on the producer and use transactions for consume-process-produce patterns. See delivery semantics for detailed trade-offs. In practice, many teams opt for at-least-once delivery with idempotent consumers as a simpler approach.

When should I use Kafka vs a simple message queue like SQS?

Use Kafka when you need event replay, multiple consumer groups reading the same data, high throughput, or stream processing. Use SQS when you need a simple, managed work queue with minimal operational overhead, especially for task distribution where messages are consumed once and discarded. Kafka is an event streaming platform; SQS is a queue.