Pub/Sub Pattern: Building Scalable Event-Driven Systems with Publish/Subscribe

Pub/Sub Pattern: Building Scalable Event-Driven Systems with Publish/Subscribe

The Publish/Subscribe (Pub/Sub) pattern is a messaging paradigm where message senders (publishers) do not send messages directly to specific receivers (subscribers). Instead, publishers categorize messages into topics without knowledge of which subscribers, if any, will receive them. Subscribers express interest in one or more topics and only receive messages that match their subscriptions. This complete decoupling of publishers and subscribers is what makes Pub/Sub the foundation of modern event-driven architectures.

How Pub/Sub Works

The Pub/Sub model has three core components:

1. Publisher: Produces messages and sends them to a topic. The publisher has no awareness of subscribers.
2. Topic (Channel): A named category that acts as a message conduit. Messages published to a topic are delivered to all subscribers of that topic.
3. Subscriber: Registers interest in one or more topics and receives all messages published to those topics.

# Conceptual Pub/Sub flow
#
# Publisher: Order Service
#   |
#   v
# Topic: "order.created"
#   |
#   +---> Subscriber: Payment Service    (processes payment)
#   +---> Subscriber: Inventory Service  (reduces stock)
#   +---> Subscriber: Email Service      (sends confirmation)
#   +---> Subscriber: Analytics Service  (tracks metrics)
#
# The Order Service publishes ONE event.
# Four services receive it independently — this is fan-out.

Fan-Out: The Power of Pub/Sub

Fan-out is the defining characteristic of Pub/Sub. A single published message is delivered to all subscribers. This is fundamentally different from a point-to-point message queue where each message goes to one consumer.

Pub/Sub Implementations

Redis Pub/Sub

# Redis Pub/Sub — simplest implementation

# Publisher
import redis
r = redis.Redis()
r.publish('order.created', '{"order_id": "ORD-5001", "total": 59.98}')

# Subscriber
pubsub = r.pubsub()
pubsub.subscribe('order.created')
pubsub.psubscribe('order.*')  # Pattern subscription

for message in pubsub.listen():
    if message['type'] == 'message':
        print(f"Channel: {message['channel']}, Data: {message['data']}")

Redis Pub/Sub is fire-and-forget — if a subscriber is offline, it misses messages permanently. No persistence, no replay. Use it for real-time notifications where message loss is acceptable (like cache invalidation broadcasts).

Google Cloud Pub/Sub

from google.cloud import pubsub_v1
import json

# Publisher
publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path('my-project', 'order-events')

data = json.dumps({'order_id': 'ORD-5001', 'total': 59.98}).encode('utf-8')
future = publisher.publish(topic_path, data, event_type='order.created')
print(f"Published message ID: {future.result()}")

# Subscriber
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path('my-project', 'payment-sub')

def callback(message):
    order = json.loads(message.data.decode('utf-8'))
    process_payment(order)
    message.ack()  # Acknowledge after processing

streaming_pull = subscriber.subscribe(subscription_path, callback=callback)
print("Listening for messages...")

Google Cloud Pub/Sub is a fully managed service with at-least-once delivery, message retention (up to 31 days), and automatic scaling. Each subscription gets its own independent copy of messages.

Amazon SNS (Simple Notification Service)

import boto3

sns = boto3.client('sns')

# Create topic
response = sns.create_topic(Name='order-events')
topic_arn = response['TopicArn']

# Subscribe — SNS supports multiple protocol endpoints
sns.subscribe(TopicArn=topic_arn, Protocol='sqs', Endpoint=sqs_queue_arn)
sns.subscribe(TopicArn=topic_arn, Protocol='lambda', Endpoint=lambda_arn)
sns.subscribe(TopicArn=topic_arn, Protocol='email', Endpoint='ops@example.com')

# Publish
sns.publish(
    TopicArn=topic_arn,
    Message=json.dumps({'order_id': 'ORD-5001'}),
    MessageAttributes={
        'event_type': {'DataType': 'String', 'StringValue': 'order.created'}
    }
)

SNS is often paired with SQS (SNS fan-out to SQS queues) to combine Pub/Sub fan-out with durable queue processing — the SNS-SQS pattern.

Apache Kafka as Pub/Sub

Kafka naturally supports Pub/Sub through consumer groups. When multiple consumer groups subscribe to the same topic, each group receives all messages independently — this is Pub/Sub with the added benefits of message retention and replay.

Pub/Sub Patterns

Event Notification

The simplest pattern: publish an event when something happens, let interested services react. The publisher does not care who listens or what they do with the event.

Event-Carried State Transfer

The event message carries the full state of the changed entity, so subscribers do not need to call back to the publisher for details. This reduces coupling and network calls but increases message size.

Message Filtering

Subscribers only receive messages matching specific criteria. This can be implemented with topic-based filtering (subscribe to specific topics) or attribute-based filtering (filter on message metadata).

# SNS message filtering — subscriber only gets matching messages
sns.subscribe(
    TopicArn=topic_arn,
    Protocol='sqs',
    Endpoint=premium_orders_queue_arn,
    Attributes={
        'FilterPolicy': json.dumps({
            'order_tier': ['premium'],
            'total': [{'numeric': ['>=', 100]}]
        })
    }
)

Real-World Use Cases

Microservice Event Broadcasting: When a user updates their profile, the User Service publishes a "user.updated" event. The Search Service updates its index, the Notification Service updates push tokens, and the Cache Service invalidates stale entries — all without the User Service knowing about any of them.

Real-Time Notifications: A chat application uses Pub/Sub to deliver messages to all connected clients. When a user sends a message, it is published to a channel. All clients subscribed to that channel receive the message in real-time via WebSocket.

Cross-Region Data Replication: Google Cloud Pub/Sub replicates events across regions. A service in US-East publishes an event; subscribers in EU-West and Asia-East receive it, enabling globally consistent data processing.

IoT Sensor Data: Thousands of IoT devices publish sensor readings to topics organized by device type or location. Analytics services subscribe to process data, monitoring services subscribe for alerts, and storage services subscribe for archival.

Challenges and Solutions

• Message ordering: Pub/Sub systems generally do not guarantee message ordering across subscribers. Solutions include using ordering keys (Google Pub/Sub) or Kafka partitions.
• Exactly-once processing: Most Pub/Sub systems provide at-least-once delivery. Design subscribers to be idempotent — processing the same message twice should not cause harm.
• Subscriber lag: A slow subscriber can fall behind. Use dead letter queues for messages that cannot be processed and monitor subscriber lag metrics.
• Fan-out cost: With many subscribers, fan-out multiplies message volume. A single published message becomes N messages (one per subscriber). Design for this cost in capacity planning.

Frequently Asked Questions

When should I use Pub/Sub vs a direct API call?

Use Pub/Sub when the publisher does not need an immediate response, when multiple services need to react to the same event, or when you want to add new consumers without modifying the producer. Use direct API calls when you need synchronous request-response (e.g., user authentication, data queries) or when the caller needs to know the result immediately.

How do I handle subscriber failures in Pub/Sub?

Managed Pub/Sub services (Google Pub/Sub, SNS+SQS) retain unacknowledged messages and redeliver them. Configure an acknowledgment deadline (how long the system waits before redelivery) and a maximum retry count. After max retries, route to a dead letter queue. For Redis Pub/Sub, which has no retry mechanism, pair it with a durable queue for critical messages.

Can I have both fan-out and competing consumers?

Yes — this is a common pattern. Use Pub/Sub for fan-out to multiple service types, then each service type uses a shared queue with competing consumers for load balancing. In AWS, this is the SNS → SQS pattern. In Kafka, this happens naturally: different consumer groups provide fan-out, while multiple consumers within a group provide competing consumption.

What is the difference between Pub/Sub and the Observer pattern?

The Observer pattern is an in-process design pattern where objects directly notify their observers. Pub/Sub is a distributed messaging pattern where a broker mediates between publishers and subscribers. Pub/Sub provides network-level decoupling, durability, and scalability that the Observer pattern does not. Think of Pub/Sub as the distributed, durable version of the Observer pattern.