Architecture Questions

Perfect Answer

Cassandra uses a peer-to-peer ring architecture where:

• No Single Point of Failure: All nodes are equal - no master/slave
• Token Ring: 0 to 2^63-1 distributed among nodes
• Consistent Hashing: Each node owns a range of tokens
• Virtual Nodes (Vnodes): Each physical node owns multiple token ranges for better distribution

Advantages over Master-Slave:

• ✅ No Bottleneck: Any node can handle any request
• ✅ True High Availability: No master election downtime
• ✅ Linear Scalability: Just add more nodes to the ring
• ✅ Automatic Rebalancing: Vnodes distribute data evenly
• ✅ Geographical Distribution: Nodes across datacenters equally

Example: Netflix uses Cassandra's ring architecture to handle 1 trillion requests/day across global datacenters without any single point of failure.

Perfect Answer

Vnodes divide each physical node into multiple virtual nodes (256 by default), each owning a small token range.

Benefits:

• 📊 Balanced Distribution: Data spreads evenly across cluster
• ⚡ Faster Rebuilds: Failed node's data distributed across many nodes
• 🔄 Easier Scaling: Adding/removing nodes rebalances automatically
• 💪 Heterogeneous Hardware: Powerful nodes get more vnodes

Without Vnodes: If Node A fails, only Node B (next in ring) rebuilds. Slow!

With Vnodes: All nodes participate in rebuild. Much faster!

Perfect Answer

Gossip is a peer-to-peer communication protocol for cluster state sharing.

How it works:

• ⏰ Every second: Each node picks 1-3 random nodes
• 💬 Exchanges state: Share node health, schema, token ownership
• 🔄 Epidemic spread: Information propagates exponentially
• 📊 Convergence: Entire cluster learns state in O(log N) rounds

Information Shared:

• Node status (UP, DOWN, JOINING, LEAVING)
• Schema versions
• Load information
• Token ownership

Why needed: In a peer-to-peer system with no central coordinator, gossip ensures eventual consistency of cluster metadata across all nodes.

Example: When Node A goes down, within 3-5 seconds, all nodes know via gossip and stop routing requests to it.

Perfect Answer

Replication Factor (RF) = Number of copies of each data row across the cluster.

Common Configurations:

• RF=1: ❌ No redundancy - data loss if node fails
• RF=2: ⚠️ Minimum for dev - survives 1 node failure
• RF=3: ✅ Production standard - survives 2 node failures
• RF=5: 🏢 High availability - survives 4 node failures

Selection Criteria:

• 💰 Cost vs Availability: RF=3 is sweet spot (3x storage)
• 📊 Read Performance: Higher RF = more replicas to read from
• 🔧 Maintenance: Higher RF = easier operations (can take nodes down)
• 🌍 Multi-DC: RF=3 per datacenter typically

Formula for fault tolerance: Can lose (RF - 1) nodes and still serve reads/writes with QUORUM

Perfect Answer

Hinted Handoff ensures data isn't lost when a replica node is temporarily down.

How it works:

1. Write arrives for partition owned by Node A, B, C (RF=3)
2. Node B is temporarily down
3. Coordinator writes to A and C successfully
4. Coordinator stores a "hint" on another node (Node D)
5. When Node B comes back up, Node D replays the hint to B
6. Node B is now up-to-date

Important Details:

• ⏰ Hint timeout: 3 hours by default (max_hint_window)
• 💾 Storage: Hints stored on local disk
• 🔄 Not counted: Hints don't count toward consistency level
• ⚠️ Limitation: If node down >3 hours, run manual repair

Production Note: Hinted handoff is why temporary node failures don't cause data loss!

Perfect Answer

QUORUM = Majority of replicas must respond (RF/2 + 1 rounded down)

Why it's popular:

• ⚖️ Balance: Good consistency without sacrificing availability
• ✅ Strong Consistency: R(QUORUM) + W(QUORUM) > RF guarantees latest data
• 💪 Fault Tolerance: Works even if (RF - QUORUM) nodes are down
• 📊 Production Standard: Used by Netflix, Instagram, Uber

Example with RF=3:

• Write needs 2/3 nodes → Success even if 1 node down
• Read checks 2/3 nodes → Gets latest value (overlap guaranteed)
• Can lose 1 node and still operate normally

Perfect Answer

Cassandra lets you choose consistency level per-query, trading off consistency for availability/latency.

Consistency Spectrum:

Real-World Use Cases:

• Social Media Feed: Write(LOCAL_QUORUM) + Read(ONE) - Speed > Consistency
• Financial Transactions: Write(QUORUM) + Read(QUORUM) - Strong consistency needed
• Logging/Metrics: Write(ONE) - Speed critical, some loss acceptable
• User Profiles: Write(QUORUM) + Read(ONE) - Recent writes critical on write

Strong Consistency Formula: W + R > RF (e.g., QUORUM + QUORUM > 3)

Perfect Answer

Cassandra uses Consistent Hashing with Virtual Nodes for data distribution.

Step-by-step process:

1. Partition Key: Extract from PRIMARY KEY
2. Hash Function: Murmur3 hash (default partitioner)
3. Token: Hash produces 64-bit token (range: -2^63 to 2^63-1)
4. Ring Lookup: Token mapped to node owning that range
5. Replica Placement: RF additional nodes clockwise in ring

Key Benefits:

• 📊 Even Distribution: Hash ensures uniform spread
• 🎯 Fast Lookups: O(1) to find owning node
• ⚡ No Hotspots: Unless poor partition key choice
• 🔄 Dynamic: Easy to add/remove nodes

Perfect Answer

Hot partition = One partition receiving disproportionate traffic, overloading specific nodes.

Common Causes:

• ❌ Poor Partition Key: Low cardinality (e.g., status='active')
• ❌ Time-based: Current date/hour as partition key
• ❌ Celebrity: One user_id getting massive traffic
• ❌ Batch Jobs: Processing only recent data

Detection:

Solutions:

• ✅ Bucketing: Add bucket to partition key (e.g., (user_id, bucket))
• ✅ Random Salt: Add random suffix to spread data
• ✅ Composite Keys: Combine multiple high-cardinality columns
• ✅ Application-level: Cache hot data in Redis/Memcached

Example Fix:

Perfect Answer

Cassandra writes are fast because they're append-only with no reads required.

Write Path (4 steps):

1. Commit Log: Sequential append to disk (durability)
2. Memtable: Write to in-memory structure (speed)
3. Response: Acknowledge to client immediately
4. Flush: Memtable → SSTable periodically

Why it's fast:

• ⚡ No Read-Before-Write: Unlike RDBMS updates
• 📝 Sequential I/O: Commit log is append-only
• 💾 Memory Writes: Memtable in RAM
• 🔄 Async Flush: SSTable flush happens in background
• 🚫 No Locks: No locking or coordination needed

Performance:

• Can handle 10,000-100,000+ writes/sec per node
• Write latency typically <1ms locally
• Linear scalability - double nodes = double throughput

Perfect Answer

Reads are complex because data might be in multiple places (Memtable + multiple SSTables).

Read Path:

1. Row Cache: Check if full row cached (fastest)
2. Bloom Filter: Check each SSTable's bloom filter
3. Key Cache: Check for partition key location
4. Partition Index: Find partition in SSTable
5. Memtable: Check in-memory data
6. SSTables: Read from disk (potentially multiple)
7. Merge: Combine data using timestamp (last write wins)
8. Read Repair: Check consistency if enabled

Why slower than writes:

• 🔍 Multiple Sources: Must check memtable + SSTables
• 💿 Disk I/O: Random reads from SSTables
• 🔀 Merge: Combine data from multiple sources
• ⚖️ Consistency: May query multiple replicas

Optimizations:

• ✅ Compaction: Fewer SSTables = faster reads
• ✅ Row Cache: Cache frequently read rows
• ✅ SSD: Fast random reads
• ✅ Partition Key Queries: Always query by partition key

Perfect Answer

Compaction merges multiple SSTables into fewer, larger SSTables to improve read performance.

Why Necessary:

• 📈 SSTable Proliferation: Each memtable flush creates new SSTable
• 🐌 Read Amplification: More SSTables = slower reads
• 💀 Tombstone Removal: Deleted data needs cleanup
• 💾 Disk Space: Reclaim space from deleted/updated data

Compaction Strategies:

• SizeTieredCompactionStrategy (STCS): Default, general purpose
• LeveledCompactionStrategy (LCS): Better for read-heavy, predictable space
• TimeWindowCompactionStrategy (TWCS): Perfect for time-series data

Trade-offs:

• ⚡ CPU/Disk: Compaction uses resources
• 💾 Space: Needs 2x space during compaction
• ⏰ Timing: Can impact performance if not tuned

Perfect Answer

Cassandra continues operating normally due to replication - no single point of failure!

Immediate Response (within seconds):

1. Gossip Detection: Other nodes detect failure via gossip
2. Mark Down: Node marked as DOWN in cluster state
3. Reroute Requests: Coordinator stops sending requests to failed node
4. Serve from Replicas: Other RF-1 replicas handle all requests
5. Hinted Handoff: Hints stored for missed writes

Different Scenarios:

• Temporary (minutes): Hinted handoff replays writes when back
• Short-term (hours): Hinted handoff up to 3 hours
• Long-term (>3 hours): Run `nodetool repair` to sync data
• Permanent: Replace node, stream data from replicas

Impact on Operations:

• 📖 Reads: No impact if RF > consistency level
• ✍️ Writes: No impact if RF > consistency level
• ⚖️ QUORUM: Works as long as majority still up

Example: RF=3, QUORUM writes. One node fails → 2 replicas still available → Writes continue normally!

Perfect Answer

Read Repair fixes inconsistencies between replicas detected during reads.

How it works:

1. Client reads with QUORUM (2/3 replicas respond)
2. Coordinator compares timestamps from all replicas
3. If mismatch detected, coordinator returns latest version to client
4. Coordinator asynchronously updates stale replicas with latest data

Two Types:

• Blocking Read Repair: Checks all replicas before responding (slower but consistent)
• Background Read Repair: Returns immediately, repairs async (faster)

When it's needed:

• Node was down during writes (hinted handoff failed)
• Write didn't reach all replicas
• Replica corruption

Alternative: `nodetool repair` - manual repair of entire node/table