Redis: The Complete Guide to the World's Most Popular In-Memory Data Store

Redis: The Complete Guide to the World's Most Popular In-Memory Data Store

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store used as a cache, database, message broker, and streaming engine. Created by Salvatore Sanfilippo in 2009, Redis has become the default choice for caching and real-time data needs in modern architectures. Its sub-millisecond latency and rich data structure support make it indispensable for high-performance applications.

Core Data Structures

Unlike simple key-value stores like Memcached, Redis supports multiple data structures, each optimized for specific use cases.

Strings

The simplest data type — a key mapped to a value (up to 512 MB). Used for caching, counters, and simple key-value storage.

SET user:1001:name "Alice"
GET user:1001:name                    # "Alice"

# Atomic increment/decrement (great for counters)
SET page:views 0
INCR page:views                       # 1
INCRBY page:views 10                  # 11

# Set with expiration
SETEX session:abc123 1800 '{"user_id":1001}'  # Expires in 30 min
SET api:key:rate 0 EX 60              # Expires in 60 seconds

# Set only if key does not exist (distributed lock)
SETNX lock:order:5001 "worker-1"     # Returns 1 (success) or 0 (already exists)

Hashes

Maps of field-value pairs — perfect for representing objects without serialization overhead.

HSET user:1001 name "Alice" email "alice@example.com" role "admin"
HGET user:1001 name                   # "Alice"
HGETALL user:1001                     # Returns all fields and values
HMGET user:1001 name email            # Get multiple fields at once
HINCRBY user:1001 login_count 1       # Atomic field increment
HDEL user:1001 role                   # Delete a single field

Lists

Ordered sequences of strings. Used for queues, recent activity feeds, and bounded collections.

LPUSH notifications:1001 "New follower: Bob"
LPUSH notifications:1001 "Your post got 100 likes"
LRANGE notifications:1001 0 9        # Get latest 10 notifications
LTRIM notifications:1001 0 99        # Keep only last 100 items
RPOP notifications:1001              # Remove and return oldest
LLEN notifications:1001              # Count of items

Sets

Unordered collections of unique strings. Useful for tags, unique visitors, and set operations.

SADD product:1001:tags "electronics" "sale" "featured"
SMEMBERS product:1001:tags            # All tags
SISMEMBER product:1001:tags "sale"    # 1 (true) or 0 (false)
SINTER product:1001:tags product:1002:tags  # Common tags
SCARD online:users                    # Count of unique online users

Sorted Sets

Like sets but each member has a score. Elements are ordered by score. Perfect for leaderboards, priority queues, and time-series indexing.

# Leaderboard
ZADD leaderboard 1500 "alice" 1200 "bob" 1800 "charlie"
ZRANGE leaderboard 0 -1 WITHSCORES          # Ascending order
ZREVRANGE leaderboard 0 2 WITHSCORES        # Top 3 players
ZRANK leaderboard "alice"                    # Alice's rank (0-indexed)
ZINCRBY leaderboard 50 "bob"                 # Bob scores 50 more points

# Time-series: recent events by timestamp
ZADD events:user:1001 1700000000 "login"
ZADD events:user:1001 1700003600 "purchase"
ZRANGEBYSCORE events:user:1001 1700000000 1700086400  # Events in time range

Streams

An append-only log data structure introduced in Redis 5.0. Streams support consumer groups, making Redis a lightweight alternative to Kafka for certain use cases.

HyperLogLog

A probabilistic data structure for counting unique elements with constant memory (12 KB regardless of cardinality). Useful for counting unique page visitors or unique search queries.

PFADD daily:visitors:2024-01-15 "user:1001" "user:1002" "user:1001"
PFCOUNT daily:visitors:2024-01-15    # 2 (approximate unique count)
PFMERGE weekly:visitors daily:visitors:2024-01-15 daily:visitors:2024-01-16

Pub/Sub Messaging

Redis includes a built-in publish/subscribe messaging system for real-time communication between services.

# Subscriber (Terminal 1)
SUBSCRIBE notifications:user:1001
PSUBSCRIBE notifications:*          # Pattern-based subscription

# Publisher (Terminal 2)
PUBLISH notifications:user:1001 "You have a new message"

# In application code (Python)
import redis

r = redis.Redis()
pubsub = r.pubsub()
pubsub.subscribe("cache:invalidate")

for message in pubsub.listen():
    if message["type"] == "message":
        key = message["data"].decode()
        print(f"Invalidating cache key: {key}")

Note: Redis Pub/Sub is fire-and-forget — messages are not persisted. If a subscriber is offline, it misses the message. For durable messaging, use Redis Streams or a dedicated message queue.

Persistence: RDB vs AOF

Redis is in-memory but supports persistence to survive restarts.

# redis.conf — Persistence configuration

# RDB: Save snapshot if at least N changes in M seconds
save 900 1        # Save if 1+ changes in 15 minutes
save 300 10       # Save if 10+ changes in 5 minutes
save 60 10000     # Save if 10000+ changes in 1 minute

# AOF: Enable append-only file
appendonly yes
appendfsync everysec   # Options: always, everysec, no

# Hybrid (recommended): Use both RDB + AOF
aof-use-rdb-preamble yes

The recommended production configuration is the hybrid approach: AOF with RDB preamble. This gives fast recovery (RDB) with minimal data loss (AOF).

Redis Cluster and High Availability

Redis Sentinel

Sentinel provides automatic failover for Redis. It monitors master and replica instances, detects failures, and promotes a replica to master when needed.

Redis Cluster

Redis Cluster provides automatic sharding across multiple Redis nodes. Data is split into 16,384 hash slots, distributed across nodes. This enables horizontal scaling for distributed caching workloads.

# Hash slot calculation
# Redis uses CRC16(key) mod 16384 to determine the slot
# Keys with the same hash tag {tag} go to the same slot

SET {user:1001}:profile "..."    # Same slot as {user:1001}:orders
SET {user:1001}:orders "..."     # Hash tags enable multi-key operations

# Cluster info
CLUSTER INFO
CLUSTER NODES
CLUSTER SLOTS

Practical Use Cases with Code

Rate Limiter

import redis
import time

r = redis.Redis()

def is_rate_limited(user_id, max_requests=100, window_seconds=60):
    key = f"rate:{user_id}:{int(time.time()) // window_seconds}"
    
    current = r.incr(key)
    if current == 1:
        r.expire(key, window_seconds)
    
    return current > max_requests

# Usage
if is_rate_limited("user:1001"):
    return "429 Too Many Requests"

Distributed Lock

import redis
import uuid

r = redis.Redis()

def acquire_lock(lock_name, timeout=10):
    token = str(uuid.uuid4())
    acquired = r.set(f"lock:{lock_name}", token, nx=True, ex=timeout)
    return token if acquired else None

def release_lock(lock_name, token):
    # Lua script ensures atomic check-and-delete
    lua_script = """
    if redis.call("GET", KEYS[1]) == ARGV[1] then
        return redis.call("DEL", KEYS[1])
    else
        return 0
    end
    """
    r.eval(lua_script, 1, f"lock:{lock_name}", token)

Redis vs Memcached

Pipelining and Transactions

Pipelining sends multiple commands without waiting for individual responses, dramatically reducing network round trips:

# Python pipeline example
pipe = r.pipeline()
for i in range(1000):
    pipe.set(f"key:{i}", f"value:{i}")
pipe.execute()  # One round trip instead of 1000

# Transaction with MULTI/EXEC
MULTI
SET account:alice:balance 900
SET account:bob:balance 1100
EXEC    # Both commands execute atomically

Frequently Asked Questions

Is Redis single-threaded? How can it be so fast?

Redis uses a single thread for command processing, which eliminates lock contention and context switching overhead. It achieves high performance through efficient in-memory data structures, non-blocking I/O multiplexing (epoll/kqueue), and keeping operations simple and fast. A single Redis instance can handle 100,000+ operations per second. Redis 6.0 introduced I/O threads for network read/write, while command execution remains single-threaded.

When should I use Redis Streams instead of Kafka?

Use Redis Streams for lighter messaging workloads where you already have Redis in your stack and do not need Kafka's massive throughput (millions of messages per second) or long-term retention. Redis Streams are excellent for real-time notifications, activity feeds, and small-scale event-driven systems. Use Kafka when you need durable, high-throughput event streaming with multi-day retention.

How do I handle hot keys in Redis?

Hot keys — keys accessed far more frequently than others — can overload a single Redis node. Solutions include adding a local in-process cache (L1 cache) in front of Redis, replicating hot keys across multiple read replicas, or splitting the key into multiple sub-keys with a random suffix and aggregating results client-side.

Should I use Redis as my primary database?

Redis can serve as a primary database for specific use cases (session storage, real-time leaderboards, rate limiting) where the data fits in memory and eventual consistency is acceptable. For general-purpose data storage with complex queries, transactions, and data larger than available RAM, use a traditional database with Redis as a cache-aside layer.

What are the memory optimization techniques for Redis?

Use appropriate data structures (hashes for small objects instead of individual string keys), set maxmemory with an appropriate eviction policy, compress large values before storing, use short key names in high-volume scenarios, and monitor memory usage with INFO MEMORY and MEMORY USAGE key commands.