Design Twitter: A Social Media Microblogging Platform

Design Twitter: A Social Media Microblogging Platform

Twitter (now X) processes hundreds of millions of tweets daily and serves billions of timeline reads. It is a classic system design interview question because of the fan-out problem, timeline generation challenges, the celebrity problem, and trending topic computation. This guide walks through designing Twitter's core systems from scratch.

1. Requirements

Functional Requirements

• Users can post tweets (up to 280 characters) with optional media.
• Users can follow other users (asymmetric relationship).
• Home timeline: a feed of tweets from followed users, ordered by relevance or time.
• User timeline: all tweets posted by a specific user.
• Like, retweet, and reply to tweets.
• Search tweets by keywords and hashtags.
• Trending topics: real-time computation of popular hashtags and topics.
• Notifications for mentions, likes, retweets, and new followers.

Non-Functional Requirements

• High availability: 99.99% uptime.
• Low latency: Timeline loads in under 300ms.
• Scalability: 400M+ DAU, 500M+ tweets/day.
• Eventual consistency: Timelines can tolerate seconds of delay.
• Extremely read-heavy (timeline reads >> tweet writes).

2. Capacity Estimation

The fan-out write volume (1.16M inserts/sec into timeline caches) is the core scaling challenge. For celebrities with millions of followers, fan-out on write is impractical.

3. High-Level Design

4. Detailed Component Design

4.1 Tweet Creation and Storage

POST /api/v1/tweets
Body: {
    "text": "Hello world! #firsttweet",
    "media_ids": ["media_123"],
    "reply_to": null
}
Response: {
    "tweet_id": "1750123456789",
    "created_at": "2025-01-25T12:00:00Z"
}

When a tweet is created: (1) Store in the tweets database. (2) Publish to Kafka for fan-out. (3) Index in Elasticsearch for search. (4) Extract hashtags for trending computation. (5) Store any media references.

4.2 Fan-out Strategy: The Core Problem

This is the most critical design decision. Twitter considered three approaches:

Fan-out on Write (Push)

When a user tweets, push the tweet_id into every follower's home timeline cache (Redis sorted set, scored by timestamp).

async function fanOutOnWrite(tweet) {
    const followers = await socialGraph.getFollowers(tweet.userId);

    // Push tweet to each follower's timeline
    const pipeline = redis.pipeline();
    for (const followerId of followers) {
        pipeline.zadd(`timeline:${followerId}`, tweet.createdAt, tweet.id);
        pipeline.zremrangebyrank(`timeline:${followerId}`, 0, -801);
    }
    await pipeline.exec();
}

Problem: If a celebrity with 50 million followers tweets, this generates 50 million Redis writes. A single tweet could take minutes to fan out completely.

Fan-out on Read (Pull)

When a user opens their timeline, fetch the latest tweets from all users they follow and merge them.

async function fanOutOnRead(userId) {
    const following = await socialGraph.getFollowing(userId);

    const tweetLists = await Promise.all(
        following.map(uid => tweetService.getRecentTweets(uid, limit: 20))
    );

    // Merge and sort by timestamp
    return mergeKSorted(tweetLists).slice(0, 20);
}

Problem: If a user follows 500 people, loading the timeline requires 500 database queries and a merge sort. At 70K timeline reads/sec, this is extremely expensive.

Hybrid Approach (Twitter's Solution)

Twitter uses a hybrid: fan-out on write for regular users, fan-out on read for celebrities. The threshold is approximately 10,000 followers.

async function getHomeTimeline(userId, cursor) {
    // 1. Get pre-computed timeline (from fan-out on write)
    const cachedTweetIds = await redis.zrevrangebyscore(
        `timeline:${userId}`, '+inf', cursor, 'LIMIT', 0, 20
    );

    // 2. Get followed celebrities (not fanned out)
    const celebrities = await socialGraph.getFollowedCelebrities(userId);

    // 3. Fetch recent tweets from each celebrity
    const celeTweets = await Promise.all(
        celebrities.map(c => tweetService.getRecent(c.id, since: cursor, limit: 5))
    );

    // 4. Merge celebrity tweets with cached timeline
    const allTweets = merge(cachedTweetIds, celeTweets.flat());
    allTweets.sort((a, b) => b.createdAt - a.createdAt);

    // 5. Hydrate tweet objects (fetch full data)
    return hydrate(allTweets.slice(0, 20));
}

4.3 The Celebrity Problem

Key insight: only ~0.1% of users are celebrities (>10K followers), but they generate disproportionate fan-out load. By exempting them from fan-out on write and handling them with fan-out on read, we reduce total fan-out by approximately 99% (since most followers belong to celebrity accounts).

4.4 Trending Topics

Trending topics require real-time stream processing:

1. Extract hashtags and keywords from every tweet (published to Kafka).
2. A stream processing engine (Apache Storm, Flink, or Kafka Streams) maintains sliding-window counters for each hashtag.
3. Rank hashtags by velocity of mentions (not just total count) — a topic rising from 100 to 10,000 mentions in 10 minutes is more "trending" than a stable 50,000 mentions.
4. Filter out evergreen hashtags (#love, #happy) and spam.
5. Personalize trends by location and user interests.

// Trending computation with Count-Min Sketch for memory efficiency
class TrendingComputer {
    constructor() {
        this.currentWindow = new CountMinSketch();
        this.previousWindow = new CountMinSketch();
        this.windowDuration = 300; // 5 minutes
    }

    processTweet(tweet) {
        const hashtags = extractHashtags(tweet.text);
        for (const tag of hashtags) {
            this.currentWindow.increment(tag);
        }
    }

    getTrending(country, limit = 10) {
        const candidates = this.currentWindow.getTopK(1000);

        // Score by velocity: current_count / previous_count
        const scored = candidates.map(tag => ({
            hashtag: tag,
            currentCount: this.currentWindow.estimate(tag),
            velocity: this.currentWindow.estimate(tag) /
                      Math.max(this.previousWindow.estimate(tag), 1)
        }));

        // Filter and rank
        return scored
            .filter(t => !isEvergreen(t.hashtag))
            .sort((a, b) => b.velocity - a.velocity)
            .slice(0, limit);
    }
}

4.5 Search

Tweet search uses an inverted index (Elasticsearch). Every tweet is indexed on creation. The search service supports:

• Keyword search across tweet text.
• Hashtag search (exact match on hashtag field).
• User mentions search.
• Filters: time range, user, language, engagement threshold.
• Results ranked by relevance (text match + engagement + recency).

5. Database Schema

CREATE TABLE users (
    id BIGINT PRIMARY KEY,
    username VARCHAR(15) UNIQUE NOT NULL,
    display_name VARCHAR(50),
    bio VARCHAR(160),
    follower_count INT DEFAULT 0,
    following_count INT DEFAULT 0,
    is_celebrity BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE tweets (
    id BIGINT PRIMARY KEY,
    user_id BIGINT NOT NULL,
    text VARCHAR(280) NOT NULL,
    reply_to_tweet_id BIGINT,
    retweet_of_id BIGINT,
    like_count INT DEFAULT 0,
    retweet_count INT DEFAULT 0,
    reply_count INT DEFAULT 0,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_tweets_user ON tweets(user_id, created_at DESC);

CREATE TABLE follows (
    follower_id BIGINT NOT NULL,
    followee_id BIGINT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (follower_id, followee_id)
);

CREATE INDEX idx_follows_followee ON follows(followee_id);

CREATE TABLE likes (
    user_id BIGINT NOT NULL,
    tweet_id BIGINT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, tweet_id)
);

CREATE INDEX idx_likes_tweet ON likes(tweet_id);

6. Key Trade-offs

7. Scaling Considerations

7.1 Timeline Cache Scaling

With 400M users and 800 tweet IDs per timeline (8 bytes each): 400M × 800 × 8B = 2.56 TB of Redis memory. Across replicas and overhead, approximately 10 TB of Redis cluster. Use caching strategies like evicting inactive users' timelines and rebuilding on demand.

7.2 Database Sharding

Shard tweets by tweet_id (time-based Snowflake IDs enable range-based sharding). Shard the follows table by follower_id for efficient "who do I follow?" queries. The social graph can also be stored in a graph database (like FlockDB, which Twitter built).

7.3 Search Scaling

With 500M tweets/day, the search index grows rapidly. Use time-based index rotation: create a new Elasticsearch index each day, search across recent indexes, and archive older ones. This makes deletion and compaction manageable.

7.4 Fan-out Service Scaling

The Fan-out Service must handle 1M+ Redis writes/sec. Partition fan-out workers by followee's follower range. Use Kafka to buffer tweet events and process asynchronously. During celebrity tweets, only the celebrity's user timeline is updated (not fanned out).

Use swehelper.com tools to practice fan-out calculations and timeline design.

8. Frequently Asked Questions

Q1: How does Twitter handle the celebrity problem?

Twitter marks users with more than a certain follower threshold (roughly 10K) as celebrities. When celebrities tweet, their tweets are NOT fanned out to followers' timelines. Instead, when a user reads their timeline, the Timeline Service merges the pre-computed timeline (from regular users' fan-out) with real-time fetches of recent tweets from the celebrities they follow. This reduces fan-out writes by ~99%.

Q2: How does Snowflake ID generation work?

Snowflake generates 64-bit unique IDs composed of: 41 bits for timestamp (milliseconds since custom epoch, good for ~69 years), 10 bits for machine/worker ID (1,024 workers), and 12 bits for sequence number (4,096 IDs per millisecond per worker). This gives ~4 million unique IDs per second per worker, with IDs that are roughly time-ordered for efficient database indexing.

Q3: How are trending topics computed in real-time?

A stream processing system (Kafka Streams or Storm) extracts hashtags from every tweet in real-time. It maintains sliding-window counters (typically 5-minute and 1-hour windows). Trending is based on velocity (rate of increase) rather than absolute count, so a rapidly rising topic trends even if its total count is lower than a stable popular hashtag. Results are filtered for spam and evergreen topics, then personalized by location.

Q4: What database does Twitter use for tweets?

Twitter uses a sharded MySQL cluster for tweet storage (they call it Manhattan, their custom distributed database built on top of it). Tweets are sharded by tweet_id. The home timeline is stored in Redis (sorted sets of tweet IDs). User timelines are served directly from the tweet table partitioned by user_id. See SQL vs NoSQL for context.