Real-World Project

Social Media Platform

Build Twitter/Instagram at scale with Cassandra!

📖 Project Overview

Build a real-world social media platform!

The Challenge: SocialHub - Twitter Meets Instagram

SocialHub is building the next big social network. They need a database that can handle:

  • 👥 Users: 500 million active users
  • 📝 Posts: 500 million posts/day
  • 💬 Comments: 2 billion comments/day
  • ❤️ Likes: 10 billion likes/day
  • 👤 Follow relationships: Billions of connections
  • 📰 Timeline feeds: Real-time updates

Scale Requirements

Active Users: 500 million monthly, 200 million daily
Posts/Second: ~5,800 writes/second (500M posts/day)
Likes/Second: ~115K writes/second (10B likes/day)
Timeline Reads: 1 million reads/second (peak)
Follow Graph: 50 billion relationships (avg 100 follows/user)
Storage Growth: 500M posts/day × 1KB = 500GB/day

Why Cassandra?

  • ✅ Write-optimized: Handle billions of likes/comments
  • ✅ Timeline feeds: Fan-out pattern with partitioning
  • ✅ High availability: Zero downtime for viral moments
  • ✅ Linear scalability: Add nodes as users grow
  • ✅ Geo-distribution: Global user base

🎯 Requirements & Use Cases

What we need to build!

📝

Posts & Content

  • Create post (text, images, video)
  • Get post details
  • Delete post
  • Get user's posts
  • Get posts by hashtag
📰

Timeline & Feed

  • Get home timeline (following)
  • Get user timeline (profile)
  • Real-time feed updates
  • Pagination (infinite scroll)
  • Trending posts
👥

Social Graph

  • Follow user
  • Unfollow user
  • Get followers list
  • Get following list
  • Friend recommendations
❤️

Engagement

  • Like/unlike post
  • Comment on post
  • Get post comments
  • Get like count
  • Get engagement metrics

🗺️ Data Model Design

Social media data modeling!

Social Media Data Modeling Principles

  1. Write fan-out: Denormalize timeline for fast reads
  2. User-centric: Most queries partition by user_id
  3. Time-ordered: Cluster by timestamp DESC (newest first)
  4. Counter tables: Track likes/followers efficiently
  5. Multiple views: Same data, different access patterns

Query Patterns → Table Design

Query Pattern Access Pattern Table Design
Q1: Get post details By post_id posts_by_id
Q2: User timeline By user_id + time posts_by_user
Q3: Home timeline By user_id + time (fan-out) timeline_by_user
Q4: Post comments By post_id + time comments_by_post
Q5: User followers By user_id followers_by_user
Q6: Hashtag posts By hashtag + time posts_by_hashtag

Timeline Fan-Out Architecture

    User posts → Write to multiple tables
                        │
        ┌───────────────┼───────────────┐
        │               │               │
        ▼               ▼               ▼
  posts_by_id    posts_by_user   posts_by_hashtag
  (lookup)       (profile)       (discovery)
        │               │
        └───────┬───────┘
                │
                ▼
        Fan-out to followers
                │
        ┌───────┴───────┐
        │               │
        ▼               ▼
  timeline_by_user  timeline_by_user
  (follower 1)      (follower 2)
  
  Read timeline = Single partition query!
        

📋 Schema Design

Complete table schemas!

1

Keyspace Creation

-- Create keyspace CREATE KEYSPACE socialhub WITH replication = { 'class': 'NetworkTopologyStrategy', 'us-east': 3, 'us-west': 3, 'eu-west': 3, 'asia-pacific': 3 }; USE socialhub;
2

Posts Table (by ID)

-- Post details lookup (canonical source) CREATE TABLE posts_by_id ( post_id TIMEUUID PRIMARY KEY, user_id UUID, username TEXT, content TEXT, media_urls LIST<TEXT>, hashtags SET<TEXT>, created_at TIMESTAMP, like_count COUNTER, comment_count COUNTER, share_count COUNTER ); -- Query: Get post details SELECT * FROM posts_by_id WHERE post_id = ?;
3

User Timeline Table

-- User's own posts (profile view) CREATE TABLE posts_by_user ( user_id UUID, created_at TIMESTAMP, post_id TIMEUUID, content TEXT, media_urls LIST<TEXT>, like_count INT, comment_count INT, PRIMARY KEY (user_id, created_at, post_id) ) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC); -- Query: Get user's posts SELECT * FROM posts_by_user WHERE user_id = ? LIMIT 20;
4

Home Timeline Table (Fan-Out)

-- User's home feed (posts from people they follow) CREATE TABLE timeline_by_user ( user_id UUID, -- Viewer's ID created_at TIMESTAMP, post_id TIMEUUID, author_id UUID, -- Post author author_username TEXT, content TEXT, media_urls LIST<TEXT>, like_count INT, PRIMARY KEY (user_id, created_at, post_id) ) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC); -- Query: Get home timeline SELECT * FROM timeline_by_user WHERE user_id = ? LIMIT 50; -- CRITICAL: This is the FAN-OUT table! -- When Alice posts, we write to timeline of ALL her followers
5

Comments Table

-- Comments on a post CREATE TABLE comments_by_post ( post_id TIMEUUID, created_at TIMESTAMP, comment_id TIMEUUID, user_id UUID, username TEXT, content TEXT, like_count INT, PRIMARY KEY (post_id, created_at, comment_id) ) WITH CLUSTERING ORDER BY (created_at DESC, comment_id DESC); -- Query: Get post comments SELECT * FROM comments_by_post WHERE post_id = ? LIMIT 100;
6

Likes Table

-- Track who liked what CREATE TABLE likes_by_post ( post_id TIMEUUID, user_id UUID, username TEXT, liked_at TIMESTAMP, PRIMARY KEY (post_id, user_id) ); -- Query: Check if user liked post SELECT * FROM likes_by_post WHERE post_id = ? AND user_id = ?; -- Also need: likes_by_user (user's liked posts) CREATE TABLE likes_by_user ( user_id UUID, liked_at TIMESTAMP, post_id TIMEUUID, PRIMARY KEY (user_id, liked_at, post_id) ) WITH CLUSTERING ORDER BY (liked_at DESC);
7

Follow Graph Tables

-- Followers (who follows user X) CREATE TABLE followers_by_user ( user_id UUID, -- Person being followed follower_id UUID, -- Person following follower_username TEXT, followed_at TIMESTAMP, PRIMARY KEY (user_id, follower_id) ); -- Following (who does user X follow) CREATE TABLE following_by_user ( user_id UUID, -- Person following following_id UUID, -- Person being followed following_username TEXT, followed_at TIMESTAMP, PRIMARY KEY (user_id, following_id) ); -- Query: Get followers SELECT * FROM followers_by_user WHERE user_id = ?; -- Query: Get following SELECT * FROM following_by_user WHERE user_id = ?;
8

User Profiles Table

-- User account information CREATE TABLE users ( user_id UUID PRIMARY KEY, username TEXT, email TEXT, full_name TEXT, bio TEXT, profile_image_url TEXT, follower_count COUNTER, following_count COUNTER, post_count COUNTER, created_at TIMESTAMP, verified BOOLEAN ); -- Also need username lookup CREATE TABLE users_by_username ( username TEXT PRIMARY KEY, user_id UUID );
9

Posts by Hashtag

-- Discover posts by hashtag CREATE TABLE posts_by_hashtag ( hashtag TEXT, created_at TIMESTAMP, post_id TIMEUUID, user_id UUID, username TEXT, content TEXT, media_urls LIST<TEXT>, like_count INT, PRIMARY KEY (hashtag, created_at, post_id) ) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC); -- Query: Get posts with hashtag SELECT * FROM posts_by_hashtag WHERE hashtag = 'cassandra' LIMIT 50;

💻 Implementation

Python implementation!

Post Service

# post_service.py from cassandra.cluster import Cluster from datetime import datetime import uuid class PostService: def __init__(self, contact_points): cluster = Cluster(contact_points) self.session = cluster.connect('socialhub') def create_post(self, user_id, username, content, media_urls=[], hashtags=[]): """Create a new post""" post_id = uuid.uuid1() # TIMEUUID created_at = datetime.now() # 1. Insert into posts_by_id (canonical) self.session.execute(""" INSERT INTO posts_by_id (post_id, user_id, username, content, media_urls, hashtags, created_at) VALUES (?, ?, ?, ?, ?, ?, ?) """, (post_id, user_id, username, content, media_urls, set(hashtags), created_at)) # 2. Insert into posts_by_user (profile) self.session.execute(""" INSERT INTO posts_by_user (user_id, created_at, post_id, content, media_urls, like_count, comment_count) VALUES (?, ?, ?, ?, ?, 0, 0) """, (user_id, created_at, post_id, content, media_urls)) # 3. Insert into posts_by_hashtag (for each hashtag) for hashtag in hashtags: self.session.execute(""" INSERT INTO posts_by_hashtag (hashtag, created_at, post_id, user_id, username, content, media_urls, like_count) VALUES (?, ?, ?, ?, ?, ?, ?, 0) """, (hashtag.lower(), created_at, post_id, user_id, username, content, media_urls)) # 4. Fan-out to followers (async/background job) # self.fan_out_to_followers(user_id, post_id, created_at, username, content, media_urls) return { 'post_id': post_id, 'created_at': created_at } def get_post(self, post_id): """Get post details""" row = self.session.execute( "SELECT * FROM posts_by_id WHERE post_id = ?", (post_id,) ).one() if row: return { 'post_id': row.post_id, 'user_id': row.user_id, 'username': row.username, 'content': row.content, 'media_urls': row.media_urls, 'hashtags': list(row.hashtags), 'created_at': row.created_at, 'like_count': row.like_count, 'comment_count': row.comment_count } return None def get_user_posts(self, user_id, limit=20): """Get user's timeline (their posts)""" rows = self.session.execute( "SELECT * FROM posts_by_user WHERE user_id = ? LIMIT ?", (user_id, limit) ) return [ { 'post_id': row.post_id, 'content': row.content, 'media_urls': row.media_urls, 'created_at': row.created_at, 'like_count': row.like_count, 'comment_count': row.comment_count } for row in rows ] def get_posts_by_hashtag(self, hashtag, limit=50): """Discover posts by hashtag""" rows = self.session.execute( "SELECT * FROM posts_by_hashtag WHERE hashtag = ? LIMIT ?", (hashtag.lower(), limit) ) return [ { 'post_id': row.post_id, 'user_id': row.user_id, 'username': row.username, 'content': row.content, 'created_at': row.created_at, 'like_count': row.like_count } for row in rows ] # Usage post_service = PostService(['localhost']) # Create post post = post_service.create_post( user_id=user_id, username='alice', content='Learning Cassandra for social media! #cassandra #database', media_urls=['https://...'], hashtags=['cassandra', 'database'] )

Social Graph Service

# social_graph_service.py class SocialGraphService: def __init__(self, session): self.session = session def follow_user(self, follower_id, follower_username, following_id, following_username): """Follow a user""" followed_at = datetime.now() # 1. Add to followers_by_user (following_id's followers) self.session.execute(""" INSERT INTO followers_by_user (user_id, follower_id, follower_username, followed_at) VALUES (?, ?, ?, ?) """, (following_id, follower_id, follower_username, followed_at)) # 2. Add to following_by_user (follower_id's following) self.session.execute(""" INSERT INTO following_by_user (user_id, following_id, following_username, followed_at) VALUES (?, ?, ?, ?) """, (follower_id, following_id, following_username, followed_at)) # 3. Increment counters self.session.execute( "UPDATE users SET follower_count = follower_count + 1 WHERE user_id = ?", (following_id,) ) self.session.execute( "UPDATE users SET following_count = following_count + 1 WHERE user_id = ?", (follower_id,) ) def unfollow_user(self, follower_id, following_id): """Unfollow a user""" # Delete from both tables self.session.execute( "DELETE FROM followers_by_user WHERE user_id = ? AND follower_id = ?", (following_id, follower_id) ) self.session.execute( "DELETE FROM following_by_user WHERE user_id = ? AND following_id = ?", (follower_id, following_id) ) # Decrement counters self.session.execute( "UPDATE users SET follower_count = follower_count - 1 WHERE user_id = ?", (following_id,) ) self.session.execute( "UPDATE users SET following_count = following_count - 1 WHERE user_id = ?", (follower_id,) ) def get_followers(self, user_id, limit=100): """Get user's followers""" rows = self.session.execute( "SELECT * FROM followers_by_user WHERE user_id = ? LIMIT ?", (user_id, limit) ) return [ { 'follower_id': row.follower_id, 'follower_username': row.follower_username, 'followed_at': row.followed_at } for row in rows ] def get_following(self, user_id, limit=100): """Get users this user follows""" rows = self.session.execute( "SELECT * FROM following_by_user WHERE user_id = ? LIMIT ?", (user_id, limit) ) return [ { 'following_id': row.following_id, 'following_username': row.following_username, 'followed_at': row.followed_at } for row in rows ]

📰 Timeline & Feed

Fan-out timeline implementation!

Timeline Service (Fan-Out on Write)

# timeline_service.py class TimelineService: def __init__(self, session): self.session = session def fan_out_to_followers(self, author_id, post_id, created_at, author_username, content, media_urls): """ Fan-out post to all followers' timelines This is the MAGIC of social media at scale! """ # 1. Get all followers followers = self.get_all_followers(author_id) # 2. Insert into each follower's timeline for follower in followers: self.session.execute(""" INSERT INTO timeline_by_user (user_id, created_at, post_id, author_id, author_username, content, media_urls, like_count) VALUES (?, ?, ?, ?, ?, ?, ?, 0) """, ( follower['follower_id'], created_at, post_id, author_id, author_username, content, media_urls )) print(f"Fanned out to {len(followers)} followers") def get_home_timeline(self, user_id, limit=50): """ Get user's home timeline This is FAST - single partition query! """ rows = self.session.execute( "SELECT * FROM timeline_by_user WHERE user_id = ? LIMIT ?", (user_id, limit) ) return [ { 'post_id': row.post_id, 'author_id': row.author_id, 'author_username': row.author_username, 'content': row.content, 'media_urls': row.media_urls, 'created_at': row.created_at, 'like_count': row.like_count } for row in rows ] def get_all_followers(self, user_id): """Get ALL followers for fan-out""" # In production, use pagination or streaming rows = self.session.execute( "SELECT follower_id FROM followers_by_user WHERE user_id = ?", (user_id,) ) return [{'follower_id': r.follower_id} for r in rows] # IMPORTANT: Fan-out considerations # - For celebrities with millions of followers, fan-out is expensive # - Solution 1: Hybrid (fan-out for normal users, on-demand for celebrities) # - Solution 2: Async job queue (Kafka/RabbitMQ) # - Solution 3: Rate limit fan-out, prioritize active followers

Fan-Out Trade-offs

Approach Pros Cons Use Case
Fan-Out on Write ✅ Fast reads (single partition)
✅ Simple queries
❌ Slow writes for celebrities
❌ Storage duplication
Normal users (<10K followers)
Fan-Out on Read ✅ Fast writes
✅ No duplication
❌ Slow reads (N queries)
❌ Complex aggregation
Celebrities (millions of followers)
Hybrid ✅ Best of both worlds ❌ Complex implementation Production (Twitter/Instagram)

⚡ Performance Optimization

Production optimization!

💾

Caching Strategy

  • Redis: Timeline cache (TTL: 5 min)
  • CDN: Profile images, media
  • App cache: User profiles
  • Cache warm-up: Celebrity timelines
  • Hit rate: 95%+ on popular content
📊

Fan-Out Optimization

  • Async processing: Kafka/RabbitMQ
  • Batch writes: 1000 followers/batch
  • Celebrity handling: Hybrid approach
  • Active user priority: Fan-out to active first
  • Result: <1s for 10K followers
🔍

Query Optimization

  • Single partition: All timeline queries
  • Prepared statements: Every query
  • Batch operations: Fan-out writes
  • LIMIT clauses: Pagination (50/page)
  • Consistency: ONE for reads, QUORUM writes
📈

Storage Optimization

  • Media: S3 + CloudFront (not Cassandra)
  • TTL: Old timeline entries (30 days)
  • Compression: LZ4 (3:1 ratio)
  • Compaction: LeveledCompactionStrategy
  • Result: 500GB/day → 165GB stored

🚀 Deployment Architecture

Production deployment!

Global Social Media Architecture

            Global Load Balancer (GeoDNS)
                        │
        ┌───────────────┼───────────────┬───────────────┐
        │               │               │               │
        ▼               ▼               ▼               ▼
    US-East         US-West         EU-West         Asia-Pac
   (Primary)        (Replica)       (Replica)       (Replica)
        │               │               │               │
   ┌────┴────┐     ┌────┴────┐     ┌────┴────┐     ┌────┴────┐
   │         │     │         │     │         │     │         │
   ▼         ▼     ▼         ▼     ▼         ▼     ▼         ▼
 Redis   Cassandra Redis Cassandra Redis Cassandra Redis Cassandra
 Cache   (6 nodes) Cache (6 nodes) Cache (6 nodes) Cache (6 nodes)
   │         │       │       │       │       │       │       │
   └─────────┼───────┴───────┼───────┴───────┼───────┴───────┘
             │               │               │
             ▼               ▼               ▼
         Kafka (Fan-out)  S3 (Media)  Elasticsearch (Search)
        

Production Checklist

Component Configuration Status
Cluster Size 24 nodes (6 per region) ✅
Replication RF=3 per region (12 total) ✅
Write Throughput 115K writes/second (likes) ✅
Read Latency p99 < 20ms (with cache) ✅
Fan-out Processing Kafka + async workers ✅
Media Storage S3 + CloudFront CDN ✅
Monitoring Datadog + PagerDuty ✅
Disaster Recovery Cross-region replication ✅

🎉 Project Complete!

You built a production-ready social media platform at Twitter/Instagram scale!

🎓 What You Built:

  • 💬 Scale: 500M users, 500M posts/day, 10B likes/day
  • 🗺️ Data model: Fan-out on write for fast reads
  • 📋 Schema: 9 tables covering all social features
  • 💻 Implementation: Post, social graph, timeline services
  • 📰 Timeline feed: Single-partition queries (fast!)
  • ⚡ Optimization: Redis cache, async fan-out, CDN
  • 🚀 Deployment: Multi-region with 24 nodes

💡 Key Learnings:

  1. Fan-out on write: Denormalize for fast timeline reads
  2. Write amplification: 1 post → N follower timelines
  3. Single partition reads: Timeline = one query!
  4. Hybrid approach: Fan-out for normal, on-demand for celebrities
  5. Time-ordered clustering: DESC for newest first
  6. Counter tables: Efficient likes/followers tracking
  7. Async processing: Kafka for fan-out queue

💬 Architecture Highlights:

  • ✅ 9 tables: posts, timeline, comments, likes, followers, hashtags
  • ✅ Fan-out: Write to all follower timelines
  • ✅ Fast reads: Single partition = sub-20ms
  • ✅ Write path: Async Kafka queue for fan-out
  • ✅ Global: 4 regions, 24 nodes, RF=12
  • ✅ Scalable: Linear growth with nodes

🚀 Next Steps:

  • 🔍 Add full-text search (Elasticsearch)
  • 🤖 Implement recommendation algorithm
  • 📊 Build analytics dashboard
  • 🔔 Add real-time notifications (WebSockets)
  • 📹 Add live video streaming
  • 🛡️ Implement content moderation (ML)

🎯 You're ready to build social media at scale! 💬
Ship the next Twitter/Instagram!

Advertisement

Responsive Ad