High Traffic Systems: Designing for Viral Events and Extreme Scale

High Traffic Systems: Designing for Viral Events and Extreme Scale

High traffic systems must handle sudden, massive surges in demand — Super Bowl streaming, Black Friday e-commerce, viral social media events, or breaking news. These events can increase traffic by 10-100x within minutes. A system that handles 10,000 requests per second must suddenly handle 1,000,000. This guide covers the architecture patterns, caching strategies, and operational practices that enable systems to survive and thrive under extreme load.

Anatomy of a Traffic Spike

Graceful Degradation

When traffic exceeds capacity, degrade gracefully instead of failing completely. Serve the most important functionality first.

class GracefulDegradation:
    def __init__(self):
        self.load_level = "normal"  # normal, elevated, critical
        self.feature_flags = {
            "recommendations": True,
            "reviews": True,
            "real_time_inventory": True,
            "search_suggestions": True,
            "personalization": True,
            "analytics_tracking": True,
        }

    def update_load_level(self, current_rps, max_rps):
        ratio = current_rps / max_rps
        if ratio < 0.7:
            self.set_normal()
        elif ratio < 0.9:
            self.set_elevated()
        else:
            self.set_critical()

    def set_elevated(self):
        self.load_level = "elevated"
        self.feature_flags["analytics_tracking"] = False
        self.feature_flags["search_suggestions"] = False
        self.feature_flags["personalization"] = False

    def set_critical(self):
        self.load_level = "critical"
        self.feature_flags["recommendations"] = False
        self.feature_flags["reviews"] = False
        self.feature_flags["real_time_inventory"] = False
        # Only core shopping flow remains: browse, cart, checkout

    def set_normal(self):
        self.load_level = "normal"
        for key in self.feature_flags:
            self.feature_flags[key] = True

Queue-Based Architecture for Traffic Spikes

import redis
import json
import time

class OrderQueue:
    def __init__(self):
        self.redis = redis.Redis()
        self.queue_name = "order_queue"
        self.max_queue_size = 100000

    def submit_order(self, order_data):
        queue_size = self.redis.llen(self.queue_name)

        if queue_size > self.max_queue_size:
            return {
                "status": "queued",
                "message": "High demand. Your order is in line.",
                "position": queue_size,
                "estimated_wait": f"{queue_size // 1000} minutes"
            }

        order_id = generate_order_id()
        self.redis.rpush(self.queue_name, json.dumps({
            "order_id": order_id,
            "data": order_data,
            "submitted_at": time.time()
        }))

        return {
            "status": "queued",
            "order_id": order_id,
            "message": "Order received. Processing shortly."
        }

    def process_orders(self, batch_size=50):
        """Workers consume from queue at sustainable rate."""
        batch = []
        for _ in range(batch_size):
            item = self.redis.lpop(self.queue_name)
            if item:
                batch.append(json.loads(item))
        if batch:
            process_order_batch(batch)

Multi-Layer Caching

class MultiLayerCache:
    def __init__(self):
        self.l1_cache = {}           # In-memory (per instance)
        self.l2_cache = redis.Redis() # Distributed cache
        self.l1_ttl = 30             # 30 seconds local
        self.l2_ttl = 300            # 5 minutes Redis

    def get(self, key):
        # Layer 1: In-memory cache (fastest, per-instance)
        if key in self.l1_cache:
            entry = self.l1_cache[key]
            if entry["expires"] > time.time():
                return entry["value"]
            del self.l1_cache[key]

        # Layer 2: Distributed Redis cache
        cached = self.l2_cache.get(f"cache:{key}")
        if cached:
            value = json.loads(cached)
            self.l1_cache[key] = {
                "value": value,
                "expires": time.time() + self.l1_ttl
            }
            return value

        # Layer 3: Database (slowest)
        value = self.db.query(key)
        if value:
            self.l2_cache.setex(f"cache:{key}", self.l2_ttl,
                               json.dumps(value))
            self.l1_cache[key] = {
                "value": value,
                "expires": time.time() + self.l1_ttl
            }
        return value

Virtual Waiting Room

For known high-demand events (product launches, ticket sales), implement a virtual queue that controls the rate of users entering the site.

class VirtualWaitingRoom:
    def __init__(self, max_active_users=10000, drain_rate=100):
        self.redis = redis.Redis()
        self.max_active = max_active_users
        self.drain_rate = drain_rate  # users admitted per second

    def enter_queue(self, user_id):
        position = self.redis.rpush("waiting_room", user_id)
        active_count = self.redis.scard("active_users")

        if active_count < self.max_active:
            self.admit_user(user_id)
            return {"status": "admitted", "token": generate_access_token(user_id)}

        estimated_wait = (position - self.max_active) / self.drain_rate
        return {
            "status": "waiting",
            "position": position,
            "estimated_wait_seconds": int(estimated_wait)
        }

    def admit_user(self, user_id):
        self.redis.sadd("active_users", user_id)
        self.redis.expire(f"active:{user_id}", 1800)  # 30-min session

    def drain_queue(self):
        """Called periodically to admit waiting users."""
        active_count = self.redis.scard("active_users")
        slots_available = self.max_active - active_count
        to_admit = min(slots_available, self.drain_rate)

        for _ in range(to_admit):
            user_id = self.redis.lpop("waiting_room")
            if user_id:
                self.admit_user(user_id)

Capacity Planning

Real-World Case Studies

Black Friday at Amazon: Amazon pre-provisions infrastructure weeks ahead. They use predictive auto-scaling, multi-layer caching, and queue-based order processing. Non-essential features are pre-configured for automatic degradation.

Super Bowl Streaming: Streaming services pre-position content on CDN edge nodes, pre-warm caches, and deploy extra capacity in regions with high expected viewership. They use adaptive bitrate streaming to gracefully reduce quality under load.

Twitter During Major Events: Twitter uses a fan-out architecture where tweets from high-follower accounts are handled differently from regular tweets. Instead of fan-out-on-write (pre-delivering to all follower timelines), celebrity tweets use fan-out-on-read to avoid overwhelming the write path.

Pre-Event Checklist

• Run load tests at 3x expected peak
• Pre-scale infrastructure (do not rely solely on auto-scaling)
• Warm all caches and CDN edge nodes
• Configure graceful degradation triggers
• Set up virtual waiting rooms if applicable
• Ensure circuit breakers are configured for all dependencies
• Have rollback plans for each degradation level
• Staff an on-call war room during the event

High traffic systems combine horizontal scaling, auto-scaling, rate limiting, and performance optimization into a cohesive architecture.

Frequently Asked Questions

Q: How do I prepare for an unknown traffic spike?

You cannot predict viral events, but you can build systems that handle them gracefully. Always have auto-scaling configured with generous maximums. Implement graceful degradation at multiple levels. Use CDN for static content. Queue writes during spikes. Monitor key metrics with alerts that trigger before the system reaches capacity.

Q: Should I pre-scale or rely on auto-scaling?

For known events (product launches, sales), always pre-scale. Auto-scaling has a lag of 2-5 minutes, which is too slow for instant traffic spikes. Pre-scale to at least your expected peak 30 minutes before the event. Use auto-scaling as a safety net for unexpected demand above the pre-scaled capacity.

Q: How do I prevent database overload during traffic spikes?

Layer 1: Cache everything possible (product pages, search results, user sessions). Layer 2: Use connection pooling (PgBouncer) to limit actual database connections. Layer 3: Queue non-critical writes. Layer 4: Route reads to replicas. Layer 5: Have read-only mode as a last resort that serves cached data while the database recovers.

Q: What is the cost of building for 100x traffic spikes?

You do not need to provision 100x capacity permanently. Use cloud auto-scaling to add capacity on demand. The cost structure is: base capacity (always running) + burst capacity (pay per use during spikes) + CDN/caching (reduces origin load). The CDN and caching layers are the most cost-effective investment for handling traffic spikes.