Real-World Project
Social Media Platform
Build Twitter/Instagram at scale with Cassandra!
📖 Project Overview
Build a real-world social media platform!
The Challenge: SocialHub - Twitter Meets Instagram
SocialHub is building the next big social network. They need a database that can handle:
- 👥 Users: 500 million active users
- 📝 Posts: 500 million posts/day
- 💬 Comments: 2 billion comments/day
- ❤️ Likes: 10 billion likes/day
- 👤 Follow relationships: Billions of connections
- 📰 Timeline feeds: Real-time updates
Scale Requirements
| Active Users: | 500 million monthly, 200 million daily |
| Posts/Second: | ~5,800 writes/second (500M posts/day) |
| Likes/Second: | ~115K writes/second (10B likes/day) |
| Timeline Reads: | 1 million reads/second (peak) |
| Follow Graph: | 50 billion relationships (avg 100 follows/user) |
| Storage Growth: | 500M posts/day × 1KB = 500GB/day |
Why Cassandra?
- ✅ Write-optimized: Handle billions of likes/comments
- ✅ Timeline feeds: Fan-out pattern with partitioning
- ✅ High availability: Zero downtime for viral moments
- ✅ Linear scalability: Add nodes as users grow
- ✅ Geo-distribution: Global user base
🎯 Requirements & Use Cases
What we need to build!
Posts & Content
- Create post (text, images, video)
- Get post details
- Delete post
- Get user's posts
- Get posts by hashtag
Timeline & Feed
- Get home timeline (following)
- Get user timeline (profile)
- Real-time feed updates
- Pagination (infinite scroll)
- Trending posts
Social Graph
- Follow user
- Unfollow user
- Get followers list
- Get following list
- Friend recommendations
Engagement
- Like/unlike post
- Comment on post
- Get post comments
- Get like count
- Get engagement metrics
🗺️ Data Model Design
Social media data modeling!
Social Media Data Modeling Principles
- Write fan-out: Denormalize timeline for fast reads
- User-centric: Most queries partition by user_id
- Time-ordered: Cluster by timestamp DESC (newest first)
- Counter tables: Track likes/followers efficiently
- Multiple views: Same data, different access patterns
Query Patterns → Table Design
| Query Pattern | Access Pattern | Table Design |
|---|---|---|
| Q1: Get post details | By post_id | posts_by_id |
| Q2: User timeline | By user_id + time | posts_by_user |
| Q3: Home timeline | By user_id + time (fan-out) | timeline_by_user |
| Q4: Post comments | By post_id + time | comments_by_post |
| Q5: User followers | By user_id | followers_by_user |
| Q6: Hashtag posts | By hashtag + time | posts_by_hashtag |
Timeline Fan-Out Architecture
User posts → Write to multiple tables
│
┌───────────────┼───────────────┐
│ │ │
▼ ▼ ▼
posts_by_id posts_by_user posts_by_hashtag
(lookup) (profile) (discovery)
│ │
└───────┬───────┘
│
▼
Fan-out to followers
│
┌───────┴───────┐
│ │
▼ ▼
timeline_by_user timeline_by_user
(follower 1) (follower 2)
Read timeline = Single partition query!
📋 Schema Design
Complete table schemas!
1
Keyspace Creation
-- Create keyspace
CREATE KEYSPACE socialhub
WITH replication = {
'class': 'NetworkTopologyStrategy',
'us-east': 3,
'us-west': 3,
'eu-west': 3,
'asia-pacific': 3
};
USE socialhub;
2
Posts Table (by ID)
-- Post details lookup (canonical source)
CREATE TABLE posts_by_id (
post_id TIMEUUID PRIMARY KEY,
user_id UUID,
username TEXT,
content TEXT,
media_urls LIST<TEXT>,
hashtags SET<TEXT>,
created_at TIMESTAMP,
like_count COUNTER,
comment_count COUNTER,
share_count COUNTER
);
-- Query: Get post details
SELECT * FROM posts_by_id WHERE post_id = ?;
3
User Timeline Table
-- User's own posts (profile view)
CREATE TABLE posts_by_user (
user_id UUID,
created_at TIMESTAMP,
post_id TIMEUUID,
content TEXT,
media_urls LIST<TEXT>,
like_count INT,
comment_count INT,
PRIMARY KEY (user_id, created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC);
-- Query: Get user's posts
SELECT * FROM posts_by_user
WHERE user_id = ?
LIMIT 20;
4
Home Timeline Table (Fan-Out)
-- User's home feed (posts from people they follow)
CREATE TABLE timeline_by_user (
user_id UUID, -- Viewer's ID
created_at TIMESTAMP,
post_id TIMEUUID,
author_id UUID, -- Post author
author_username TEXT,
content TEXT,
media_urls LIST<TEXT>,
like_count INT,
PRIMARY KEY (user_id, created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC);
-- Query: Get home timeline
SELECT * FROM timeline_by_user
WHERE user_id = ?
LIMIT 50;
-- CRITICAL: This is the FAN-OUT table!
-- When Alice posts, we write to timeline of ALL her followers
5
Comments Table
-- Comments on a post
CREATE TABLE comments_by_post (
post_id TIMEUUID,
created_at TIMESTAMP,
comment_id TIMEUUID,
user_id UUID,
username TEXT,
content TEXT,
like_count INT,
PRIMARY KEY (post_id, created_at, comment_id)
) WITH CLUSTERING ORDER BY (created_at DESC, comment_id DESC);
-- Query: Get post comments
SELECT * FROM comments_by_post
WHERE post_id = ?
LIMIT 100;
6
Likes Table
-- Track who liked what
CREATE TABLE likes_by_post (
post_id TIMEUUID,
user_id UUID,
username TEXT,
liked_at TIMESTAMP,
PRIMARY KEY (post_id, user_id)
);
-- Query: Check if user liked post
SELECT * FROM likes_by_post
WHERE post_id = ? AND user_id = ?;
-- Also need: likes_by_user (user's liked posts)
CREATE TABLE likes_by_user (
user_id UUID,
liked_at TIMESTAMP,
post_id TIMEUUID,
PRIMARY KEY (user_id, liked_at, post_id)
) WITH CLUSTERING ORDER BY (liked_at DESC);
7
Follow Graph Tables
-- Followers (who follows user X)
CREATE TABLE followers_by_user (
user_id UUID, -- Person being followed
follower_id UUID, -- Person following
follower_username TEXT,
followed_at TIMESTAMP,
PRIMARY KEY (user_id, follower_id)
);
-- Following (who does user X follow)
CREATE TABLE following_by_user (
user_id UUID, -- Person following
following_id UUID, -- Person being followed
following_username TEXT,
followed_at TIMESTAMP,
PRIMARY KEY (user_id, following_id)
);
-- Query: Get followers
SELECT * FROM followers_by_user WHERE user_id = ?;
-- Query: Get following
SELECT * FROM following_by_user WHERE user_id = ?;
8
User Profiles Table
-- User account information
CREATE TABLE users (
user_id UUID PRIMARY KEY,
username TEXT,
email TEXT,
full_name TEXT,
bio TEXT,
profile_image_url TEXT,
follower_count COUNTER,
following_count COUNTER,
post_count COUNTER,
created_at TIMESTAMP,
verified BOOLEAN
);
-- Also need username lookup
CREATE TABLE users_by_username (
username TEXT PRIMARY KEY,
user_id UUID
);
9
Posts by Hashtag
-- Discover posts by hashtag
CREATE TABLE posts_by_hashtag (
hashtag TEXT,
created_at TIMESTAMP,
post_id TIMEUUID,
user_id UUID,
username TEXT,
content TEXT,
media_urls LIST<TEXT>,
like_count INT,
PRIMARY KEY (hashtag, created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC, post_id DESC);
-- Query: Get posts with hashtag
SELECT * FROM posts_by_hashtag
WHERE hashtag = 'cassandra'
LIMIT 50;
💻 Implementation
Python implementation!
Post Service
# post_service.py
from cassandra.cluster import Cluster
from datetime import datetime
import uuid
class PostService:
def __init__(self, contact_points):
cluster = Cluster(contact_points)
self.session = cluster.connect('socialhub')
def create_post(self, user_id, username, content, media_urls=[], hashtags=[]):
"""Create a new post"""
post_id = uuid.uuid1() # TIMEUUID
created_at = datetime.now()
# 1. Insert into posts_by_id (canonical)
self.session.execute("""
INSERT INTO posts_by_id
(post_id, user_id, username, content, media_urls, hashtags, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (post_id, user_id, username, content, media_urls, set(hashtags), created_at))
# 2. Insert into posts_by_user (profile)
self.session.execute("""
INSERT INTO posts_by_user
(user_id, created_at, post_id, content, media_urls, like_count, comment_count)
VALUES (?, ?, ?, ?, ?, 0, 0)
""", (user_id, created_at, post_id, content, media_urls))
# 3. Insert into posts_by_hashtag (for each hashtag)
for hashtag in hashtags:
self.session.execute("""
INSERT INTO posts_by_hashtag
(hashtag, created_at, post_id, user_id, username, content, media_urls, like_count)
VALUES (?, ?, ?, ?, ?, ?, ?, 0)
""", (hashtag.lower(), created_at, post_id, user_id, username, content, media_urls))
# 4. Fan-out to followers (async/background job)
# self.fan_out_to_followers(user_id, post_id, created_at, username, content, media_urls)
return {
'post_id': post_id,
'created_at': created_at
}
def get_post(self, post_id):
"""Get post details"""
row = self.session.execute(
"SELECT * FROM posts_by_id WHERE post_id = ?",
(post_id,)
).one()
if row:
return {
'post_id': row.post_id,
'user_id': row.user_id,
'username': row.username,
'content': row.content,
'media_urls': row.media_urls,
'hashtags': list(row.hashtags),
'created_at': row.created_at,
'like_count': row.like_count,
'comment_count': row.comment_count
}
return None
def get_user_posts(self, user_id, limit=20):
"""Get user's timeline (their posts)"""
rows = self.session.execute(
"SELECT * FROM posts_by_user WHERE user_id = ? LIMIT ?",
(user_id, limit)
)
return [
{
'post_id': row.post_id,
'content': row.content,
'media_urls': row.media_urls,
'created_at': row.created_at,
'like_count': row.like_count,
'comment_count': row.comment_count
}
for row in rows
]
def get_posts_by_hashtag(self, hashtag, limit=50):
"""Discover posts by hashtag"""
rows = self.session.execute(
"SELECT * FROM posts_by_hashtag WHERE hashtag = ? LIMIT ?",
(hashtag.lower(), limit)
)
return [
{
'post_id': row.post_id,
'user_id': row.user_id,
'username': row.username,
'content': row.content,
'created_at': row.created_at,
'like_count': row.like_count
}
for row in rows
]
# Usage
post_service = PostService(['localhost'])
# Create post
post = post_service.create_post(
user_id=user_id,
username='alice',
content='Learning Cassandra for social media! #cassandra #database',
media_urls=['https://...'],
hashtags=['cassandra', 'database']
)
Social Graph Service
# social_graph_service.py
class SocialGraphService:
def __init__(self, session):
self.session = session
def follow_user(self, follower_id, follower_username, following_id, following_username):
"""Follow a user"""
followed_at = datetime.now()
# 1. Add to followers_by_user (following_id's followers)
self.session.execute("""
INSERT INTO followers_by_user
(user_id, follower_id, follower_username, followed_at)
VALUES (?, ?, ?, ?)
""", (following_id, follower_id, follower_username, followed_at))
# 2. Add to following_by_user (follower_id's following)
self.session.execute("""
INSERT INTO following_by_user
(user_id, following_id, following_username, followed_at)
VALUES (?, ?, ?, ?)
""", (follower_id, following_id, following_username, followed_at))
# 3. Increment counters
self.session.execute(
"UPDATE users SET follower_count = follower_count + 1 WHERE user_id = ?",
(following_id,)
)
self.session.execute(
"UPDATE users SET following_count = following_count + 1 WHERE user_id = ?",
(follower_id,)
)
def unfollow_user(self, follower_id, following_id):
"""Unfollow a user"""
# Delete from both tables
self.session.execute(
"DELETE FROM followers_by_user WHERE user_id = ? AND follower_id = ?",
(following_id, follower_id)
)
self.session.execute(
"DELETE FROM following_by_user WHERE user_id = ? AND following_id = ?",
(follower_id, following_id)
)
# Decrement counters
self.session.execute(
"UPDATE users SET follower_count = follower_count - 1 WHERE user_id = ?",
(following_id,)
)
self.session.execute(
"UPDATE users SET following_count = following_count - 1 WHERE user_id = ?",
(follower_id,)
)
def get_followers(self, user_id, limit=100):
"""Get user's followers"""
rows = self.session.execute(
"SELECT * FROM followers_by_user WHERE user_id = ? LIMIT ?",
(user_id, limit)
)
return [
{
'follower_id': row.follower_id,
'follower_username': row.follower_username,
'followed_at': row.followed_at
}
for row in rows
]
def get_following(self, user_id, limit=100):
"""Get users this user follows"""
rows = self.session.execute(
"SELECT * FROM following_by_user WHERE user_id = ? LIMIT ?",
(user_id, limit)
)
return [
{
'following_id': row.following_id,
'following_username': row.following_username,
'followed_at': row.followed_at
}
for row in rows
]
📰 Timeline & Feed
Fan-out timeline implementation!
Timeline Service (Fan-Out on Write)
# timeline_service.py
class TimelineService:
def __init__(self, session):
self.session = session
def fan_out_to_followers(self, author_id, post_id, created_at,
author_username, content, media_urls):
"""
Fan-out post to all followers' timelines
This is the MAGIC of social media at scale!
"""
# 1. Get all followers
followers = self.get_all_followers(author_id)
# 2. Insert into each follower's timeline
for follower in followers:
self.session.execute("""
INSERT INTO timeline_by_user
(user_id, created_at, post_id, author_id, author_username,
content, media_urls, like_count)
VALUES (?, ?, ?, ?, ?, ?, ?, 0)
""", (
follower['follower_id'],
created_at,
post_id,
author_id,
author_username,
content,
media_urls
))
print(f"Fanned out to {len(followers)} followers")
def get_home_timeline(self, user_id, limit=50):
"""
Get user's home timeline
This is FAST - single partition query!
"""
rows = self.session.execute(
"SELECT * FROM timeline_by_user WHERE user_id = ? LIMIT ?",
(user_id, limit)
)
return [
{
'post_id': row.post_id,
'author_id': row.author_id,
'author_username': row.author_username,
'content': row.content,
'media_urls': row.media_urls,
'created_at': row.created_at,
'like_count': row.like_count
}
for row in rows
]
def get_all_followers(self, user_id):
"""Get ALL followers for fan-out"""
# In production, use pagination or streaming
rows = self.session.execute(
"SELECT follower_id FROM followers_by_user WHERE user_id = ?",
(user_id,)
)
return [{'follower_id': r.follower_id} for r in rows]
# IMPORTANT: Fan-out considerations
# - For celebrities with millions of followers, fan-out is expensive
# - Solution 1: Hybrid (fan-out for normal users, on-demand for celebrities)
# - Solution 2: Async job queue (Kafka/RabbitMQ)
# - Solution 3: Rate limit fan-out, prioritize active followers
Fan-Out Trade-offs
| Approach | Pros | Cons | Use Case |
|---|---|---|---|
| Fan-Out on Write | ✅ Fast reads (single partition) ✅ Simple queries |
❌ Slow writes for celebrities ❌ Storage duplication |
Normal users (<10K followers) |
| Fan-Out on Read | ✅ Fast writes ✅ No duplication |
❌ Slow reads (N queries) ❌ Complex aggregation |
Celebrities (millions of followers) |
| Hybrid | ✅ Best of both worlds | ❌ Complex implementation | Production (Twitter/Instagram) |
⚡ Performance Optimization
Production optimization!
Caching Strategy
- Redis: Timeline cache (TTL: 5 min)
- CDN: Profile images, media
- App cache: User profiles
- Cache warm-up: Celebrity timelines
- Hit rate: 95%+ on popular content
Fan-Out Optimization
- Async processing: Kafka/RabbitMQ
- Batch writes: 1000 followers/batch
- Celebrity handling: Hybrid approach
- Active user priority: Fan-out to active first
- Result: <1s for 10K followers
Query Optimization
- Single partition: All timeline queries
- Prepared statements: Every query
- Batch operations: Fan-out writes
- LIMIT clauses: Pagination (50/page)
- Consistency: ONE for reads, QUORUM writes
Storage Optimization
- Media: S3 + CloudFront (not Cassandra)
- TTL: Old timeline entries (30 days)
- Compression: LZ4 (3:1 ratio)
- Compaction: LeveledCompactionStrategy
- Result: 500GB/day → 165GB stored
🚀 Deployment Architecture
Production deployment!
Global Social Media Architecture
Global Load Balancer (GeoDNS)
│
┌───────────────┼───────────────┬───────────────┐
│ │ │ │
▼ ▼ ▼ ▼
US-East US-West EU-West Asia-Pac
(Primary) (Replica) (Replica) (Replica)
│ │ │ │
┌────┴────┐ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
Redis Cassandra Redis Cassandra Redis Cassandra Redis Cassandra
Cache (6 nodes) Cache (6 nodes) Cache (6 nodes) Cache (6 nodes)
│ │ │ │ │ │ │ │
└─────────┼───────┴───────┼───────┴───────┼───────┴───────┘
│ │ │
▼ ▼ ▼
Kafka (Fan-out) S3 (Media) Elasticsearch (Search)
Production Checklist
| Component | Configuration | Status |
|---|---|---|
| Cluster Size | 24 nodes (6 per region) | ✅ |
| Replication | RF=3 per region (12 total) | ✅ |
| Write Throughput | 115K writes/second (likes) | ✅ |
| Read Latency | p99 < 20ms (with cache) | ✅ |
| Fan-out Processing | Kafka + async workers | ✅ |
| Media Storage | S3 + CloudFront CDN | ✅ |
| Monitoring | Datadog + PagerDuty | ✅ |
| Disaster Recovery | Cross-region replication | ✅ |
🎉 Project Complete!
You built a production-ready social media platform at Twitter/Instagram scale!
🎓 What You Built:
- 💬 Scale: 500M users, 500M posts/day, 10B likes/day
- 🗺️ Data model: Fan-out on write for fast reads
- 📋 Schema: 9 tables covering all social features
- 💻 Implementation: Post, social graph, timeline services
- 📰 Timeline feed: Single-partition queries (fast!)
- ⚡ Optimization: Redis cache, async fan-out, CDN
- 🚀 Deployment: Multi-region with 24 nodes
💡 Key Learnings:
- Fan-out on write: Denormalize for fast timeline reads
- Write amplification: 1 post → N follower timelines
- Single partition reads: Timeline = one query!
- Hybrid approach: Fan-out for normal, on-demand for celebrities
- Time-ordered clustering: DESC for newest first
- Counter tables: Efficient likes/followers tracking
- Async processing: Kafka for fan-out queue
💬 Architecture Highlights:
- ✅ 9 tables: posts, timeline, comments, likes, followers, hashtags
- ✅ Fan-out: Write to all follower timelines
- ✅ Fast reads: Single partition = sub-20ms
- ✅ Write path: Async Kafka queue for fan-out
- ✅ Global: 4 regions, 24 nodes, RF=12
- ✅ Scalable: Linear growth with nodes
🚀 Next Steps:
- 🔍 Add full-text search (Elasticsearch)
- 🤖 Implement recommendation algorithm
- 📊 Build analytics dashboard
- 🔔 Add real-time notifications (WebSockets)
- 📹 Add live video streaming
- 🛡️ Implement content moderation (ML)
🎯 You're ready to build social media at scale! 💬
Ship the next Twitter/Instagram!
Advertisement
Responsive Ad