Cassandra Ring Topology

Answer:

Ring Topology is Cassandra's logical organization where all nodes are arranged in a circular structure with no hierarchy. Each node is equal and owns a portion of the token range.

Importance:

• No Single Point of Failure: Unlike master-slave systems, there's no central node that can crash
• Automatic Load Balancing: Data distributed evenly using consistent hashing
• Linear Scalability: Adding nodes increases capacity proportionally
• Fault Tolerance: Replication moves clockwise around ring, ensuring availability

Example: If you have 4 nodes with token ranges 0-25, 26-50, 51-75, 76-100, data with hash value 67 goes to node 3 (51-75 range), and its replicas go to nodes 4, 1 (clockwise).

Answer (Step-by-Step):

Step 1: Hash the Partition Key

• Cassandra applies a hash function (Murmur3 by default) to the partition key
• Example: Hash("user123") = 8234567123456789
• This produces a token value in the range -2^63 to +2^63

Step 2: Find Owning Node

• Move clockwise around the ring from token value
• First node with token ≥ hash value owns the data
• This node becomes the primary replica

Step 3: Determine Replicas

• Continue clockwise for additional replicas based on RF
• RF=3 means data stored on 3 consecutive nodes (clockwise)
• Wraps around if needed (after last node, goes to first)

Example: Hash value 7000 with nodes at tokens [0, 5000, 10000, 15000] → First node ≥ 7000 is node at 10000. With RF=3, replicas on nodes at 10000, 15000, and 0 (wrap-around).

Tokens: Numeric values representing positions on the ring. Each node owns a range of tokens and is responsible for data that hashes to values within that range.

Manual Token Assignment (Old Way):

• Administrator calculates and assigns specific token to each node
• Formula: Token = (2^64 / number_of_nodes) × node_position
• Configured in cassandra.yaml file
• Drawbacks: Error-prone, difficult to add/remove nodes, manual rebalancing needed

Virtual Nodes / Vnodes (Modern Way):

• Each physical node owns multiple "virtual nodes" (default: 256)
• Tokens randomly distributed across the ring
• Automatic load balancing and rebalancing
• Benefits: Perfect distribution, easy scaling, faster repairs, no manual configuration

Best Practice: Use vnodes (default) unless you have specific reasons for manual assignment (very large clusters with homogeneous hardware).

Replication Process:

1. Determine Primary Replica:

• Hash partition key to get token value
• First node clockwise from token value is primary

2. Place Additional Replicas Clockwise:

• For RF=N, continue N-1 nodes clockwise
• Each subsequent node gets a replica
• Wraps around ring (circular)

3. Replication Strategy:

• SimpleStrategy: Replicas on next N-1 nodes clockwise (single DC)
• NetworkTopologyStrategy: Specify RF per datacenter, considers racks for replica placement

Example with RF=3:

Benefits: If any node fails, data still available from remaining replicas. With RF=3, can tolerate 2 simultaneous node failures.

Key Advantages:

1. No Single Point of Failure:

• Master-slave: If master fails, entire system goes down or needs failover
• Ring: All nodes equal, any node can serve requests, node failures transparent

2. Better Scalability:

• Master-slave: Master becomes bottleneck as you add slaves
• Ring: Linear scalability - 2x nodes = 2x throughput

3. Simpler Operations:

• Master-slave: Complex failover, promotion logic, split-brain scenarios
• Ring: Add/remove nodes with simple bootstrap/decommission, no special handling

4. Load Distribution:

• Master-slave: Master handles all writes, can't distribute write load
• Ring: Writes distributed across all nodes based on token ranges

5. Geographic Distribution:

• Master-slave: Cross-region replication adds latency
• Ring: Nodes in multiple regions, users connect to nearest, local reads/writes

Trade-off: Ring topology sacrifices some consistency (eventual consistency) for availability and partition tolerance (CAP theorem). Master-slave can provide stronger consistency but at cost of availability and scalability.

Process When Adding a Node:

1. Bootstrap Phase:

• New node joins cluster and contacts seed nodes
• Receives current ring topology information
• Gets assigned tokens (or vnodes if using virtual nodes)

2. Token Assignment:

• With Vnodes: Automatically gets 256 random token positions
• Manual: Administrator assigns specific token value
• New tokens inserted into ring between existing nodes

3. Data Streaming:

• Existing nodes stream data to new node for its token ranges
• Happens in background, cluster remains operational
• New node receives data it now owns based on token assignment

4. Ring Update:

• All nodes update their ring topology view via gossip
• New node becomes available for reads/writes
• Load automatically rebalanced across cluster

Example:

Benefits:

• Zero downtime during addition
• Automatic data rebalancing
• Increased capacity and throughput
• No manual resharding needed

Consistent Hashing: A hashing technique where adding or removing nodes (servers) requires redistributing only K/N keys on average, where K is total keys and N is number of nodes.

How It Works:

• Both nodes and data are mapped to same hash space (the ring)
• Nodes placed at specific points on ring based on their tokens
• Data placed at specific points based on hash of partition key
• Each piece of data assigned to first node clockwise from its position

Why It's Important:

• Minimal Disruption: Adding/removing node only affects adjacent nodes, not entire cluster
• Even Distribution: Hash function ensures uniform data spread
• Scalability: Can scale to thousands of nodes without performance degradation
• Predictability: Always know exactly where data lives

Comparison to Simple Hashing:

Virtual Nodes Enhancement: Vnodes improve consistent hashing by giving each physical node multiple positions on ring, resulting in even better load distribution.

Automatic Failure Handling:

1. Detection (Gossip Protocol):

• Nodes periodically exchange heartbeat messages
• If node doesn't respond, marked as DOWN after timeout
• Failure info propagates through cluster via gossip

2. Request Routing:

• Coordinator stops sending requests to failed node
• Requests routed to replica nodes instead
• Happens automatically, transparent to clients

3. Hinted Handoff:

• Writes for failed node temporarily stored on healthy nodes
• "Hints" saved for failed node
• When node recovers, hints are replayed to catch it up

4. Read Repair:

• During reads, coordinator compares data from replicas
• If inconsistencies found, updates stale replicas
• Ensures eventual consistency

5. Anti-Entropy Repair:

• Periodic background process (nodetool repair)
• Compares data across all replicas
• Synchronizes any differences

Example Scenario:

Consistency Level Impact: With RF=3 and QUORUM consistency, can tolerate 1 node failure and still serve requests. With LOCAL_QUORUM, can tolerate datacenter failures.

Virtual Nodes (Vnodes): Instead of each physical node owning a single continuous token range, it owns multiple smaller, non-contiguous ranges (default: 256 vnodes per physical node).

How Vnodes Work:

• Each physical node assigned 256 random token values (configurable)
• These tokens scattered throughout the ring
• Each token represents ownership of a small range
• Data ownership distributed across many small chunks

Benefits:

1. Better Load Distribution:

• 256 random positions → more uniform distribution than single token
• Reduces probability of uneven data distribution
• No "hot spots" from poor token selection

2. Easier Operations:

• No manual token calculation needed
• Automatic in Cassandra 1.2+
• Eliminates human error in token assignment

3. Faster Rebuilds:

• When node fails, its 256 ranges distributed across many nodes
• Rebuild parallelized across entire cluster
• Much faster than single-node rebuild

4. Heterogeneous Hardware:

• Can assign more/fewer vnodes based on hardware capacity
• Powerful nodes: 512 vnodes (more data)
• Weaker nodes: 128 vnodes (less data)
• Naturally balances load based on capacity

Example:

Configuration: Set in cassandra.yaml with `num_tokens: 256` (default). For very large clusters (1000+ nodes), some use fewer vnodes (8-16) to reduce overhead.

Multi-Datacenter Ring Architecture:

1. Logical Ring Per Datacenter:

• Each datacenter has its own logical ring
• Same token space, but nodes grouped by location
• Nodes aware of datacenter topology

2. NetworkTopologyStrategy:

• Replication strategy for multi-DC deployments
• Specify RF independently per datacenter
• Example: RF=3 in US-East, RF=2 in EU-West

Configuration Example:

3. Data Placement Logic:

• Primary replica: First node clockwise from hash in any DC
• Additional replicas per DC: Next N-1 nodes clockwise in that DC
• Rack-aware: Spreads replicas across racks for fault tolerance

4. Benefits:

Geographic Distribution:

• Users connect to nearest datacenter (low latency)
• Local reads with LOCAL_QUORUM consistency
• Reduced cross-DC traffic

Disaster Recovery:

• Entire datacenter can fail without data loss
• Automatic failover to other DCs
• No manual intervention needed

Compliance:

• Data residency requirements (GDPR, etc.)
• EU data stays in EU datacenters
• Controlled replication policies

Example Scenario:

Consistency Levels:

• LOCAL_QUORUM: Majority in local DC (fast, common for reads)
• EACH_QUORUM: Majority in each DC (strong consistency across DCs)
• LOCAL_ONE: Any node in local DC (fastest, eventual consistency)

Node Decommission Process:

1. Initiate Decommission:

• Run command: `nodetool decommission` on the node
• Node begins streaming its data to other nodes
• Continues serving requests during process

2. Data Streaming:

• For each token range owned by departing node
• Data streamed to next node(s) clockwise in ring
• Ensures data remains available at RF level

3. Ring Update:

• Once streaming complete, node announces departure
• Other nodes update their ring topology via gossip
• Node removed from ring, stops serving requests

4. Load Rebalancing:

• Remaining nodes now own larger token ranges
• Load distributed evenly across remaining nodes
• No manual intervention needed

Example:

Important Notes:

• Never use `nodetool removenode` on a live node (only for dead nodes)
• Ensure cluster can handle increased load before removing
• With vnodes, data distributed across many nodes for faster rebalance
• Zero downtime during decommission

Architecture Comparison:

Cassandra Ring:

• Peer-to-Peer: All nodes equal, no special roles
• Automatic Distribution: Consistent hashing distributes data
• No Config Servers: Ring info distributed via gossip
• Any Node Entry: Client can connect to any node

MongoDB Sharded Cluster:

• Config Servers: Separate servers store metadata
• Mongos Routers: Query routers direct requests to shards
• Shard Servers: Actual data storage (replica sets)
• Manual Sharding: Define shard key, chunk ranges

Key Differences:

When to Choose Each:

• Cassandra: Need maximum availability, write-heavy, geographic distribution
• MongoDB: Need complex queries, transactions, flexible schema evolution

System Design: Global Social Media Platform

Requirements:

• 500 million users worldwide
• Store user profiles, posts, comments, likes
• Read-heavy workload (90% reads, 10% writes)
• Low latency globally
• High availability (99.99%)

Architecture Design:

1. Multi-Datacenter Deployment:

2. Data Model:

3. Ring Topology Benefits:

Data Distribution:

• Hash(user_id) distributes users evenly across 120 nodes
• Each node owns ~0.8% of total data
• No hot spots (unlike following celebrities)

Geographic Locality:

• US users connect to US datacenters (low latency)
• EU users connect to EU datacenters (GDPR compliance)
• LOCAL_QUORUM consistency for reads (fast)

Fault Tolerance:

• RF=3 per DC → can lose 1 node per DC
• Can lose entire datacenter without data loss
• Automatic failover to nearest healthy DC

4. Performance Optimizations:

• Caching: Row cache for user profiles (frequently accessed)
• Compaction: Time-window compaction for time-series data (posts)
• Consistency: LOCAL_QUORUM for reads, LOCAL_ONE for analytics
• Connection Pooling: App servers maintain connection pools to local DC

5. Scaling Strategy:

• Vertical: Add more nodes to existing DCs
• Horizontal: Add new datacenters in new regions
• Zero Downtime: Use vnodes for automatic rebalancing

Expected Performance:

Cost Efficiency: Using commodity hardware (AWS i3.2xlarge or similar), estimated $200K-300K/month for 120-node cluster vs. millions for traditional RDBMS at this scale.