Section 4: Consistency & CAP

Cassandra Consistency & Quorum

Master tunable consistency! Deep dive into consistency levels (ONE, QUORUM, ALL), quorum calculations, CAP theorem, and how to balance consistency vs. availability for your specific use case.

📖 The Story: The Multi-Branch Bank

Imagine GlobalBank with branches worldwide. How do they handle account balances when customers access from different locations?

❌ THE BAD APPROACH: Ask Every Branch Every Time

Problem: Customer checks balance → Must call ALL 1000 branches worldwide

Customer Alice in Tokyo checks balance:
├─ Call 1000 branches worldwide
├─ Branch 542 in Brazil is offline ❌
├─ Waiting... waiting... TIMEOUT!
└─ Alice can't check balance! 😡

Result:
• 2-second response time (global calls)
• ANY branch offline = query fails
• Terrible user experience!

✅ THE BRILLIANT APPROACH: Flexible Agreement Levels

Solution: Choose how many branches to ask based on importance

SCENARIO 1: Quick Balance Check (ONE branch)
Alice checks balance in Tokyo:
├─ Ask Tokyo branch only (1/3 copies)
├─ Response: $10,000
└─ Time: 5ms ✅ FAST!

SCENARIO 2: Important Transaction (QUORUM 2/3)
Alice withdraws $5,000:
├─ Ask Tokyo & Singapore branches (2/3 copies)
├─ Both confirm: $10,000 available
├─ Proceed with withdrawal
└─ Time: 50ms ✅ SAFE!

SCENARIO 3: Account Closure (ALL 3 branches)
Alice closes account:
├─ Must confirm with Tokyo, Singapore, London (ALL 3)
├─ All agree: $10,000 balance
├─ Final transfer and close
└─ Time: 200ms ✅ CERTAIN!

Why It's BRILLIANT:

  • Tunable: Choose speed vs. safety per operation
  • Available: ONE branch down? Still works!
  • Fast: Quick checks use nearest branch only
  • Safe: Important ops verify with majority
  • Flexible: Different operations, different levels!

🎯 This is EXACTLY Cassandra's Tunable Consistency!

  • Branch = Replica Node
  • 3 Branches = RF=3 (3 copies)
  • ONE branch = Consistency Level ONE
  • 2/3 branches = Consistency Level QUORUM
  • ALL branches = Consistency Level ALL
  • Balance check = Fast read (ONE)
  • Withdrawal = Safe write (QUORUM)
  • Account closure = Critical operation (ALL)

GlobalBank's flexible branches = Cassandra's tunable consistency!
Same brilliance, perfect for every use case!

⚖️ What is Consistency in Cassandra?

Understanding consistency, replication, and how Cassandra ensures data correctness.

Complete Definition

Consistency: The guarantee that all replicas of data agree on the same value at a given point in time, and how many replicas must acknowledge a read or write operation before it's considered successful.

Key Concepts:

  • Replication Factor (RF): Number of copies of data (e.g., RF=3 = 3 copies)
  • Consistency Level (CL): How many replicas must respond for success
  • Quorum: Majority of replicas (QUORUM = (RF/2) + 1)
  • Tunable: Can be set per-query (read or write)
  • Strong Consistency: Read CL + Write CL > RF
  • Eventual Consistency: All replicas will eventually agree

The Fundamental Trade-off:

  • Higher Consistency: More replicas checked → Slower, less available
  • Lower Consistency: Fewer replicas checked → Faster, more available
  • Sweet Spot: QUORUM balances both (checks majority)
✅

Strong Consistency

All replicas always agree

RF=3, Write QUORUM, Read QUORUM

Write: 2/3 replicas confirm
Read: 2/3 replicas checked
Total: 2 + 2 = 4 > 3 (RF)

Result: Always see latest value!
Overlap guaranteed!
  • Pros: Always consistent
  • Cons: Slower, less available
  • Use: Financial transactions
⚡

Eventual Consistency

Replicas may temporarily differ

RF=3, Write ONE, Read ONE

Write: 1/3 replica confirms
Read: 1/3 replica checked
Total: 1 + 1 = 2 < 3 (RF)

Result: May see stale data
Eventually consistent via repair!
  • Pros: Fastest, most available
  • Cons: Temporary inconsistency
  • Use: Social media feeds

🔺 The CAP Theorem

Understanding the fundamental trade-offs in distributed systems.

CAP Theorem: Pick Two (You Can't Have All Three!) Consistency All nodes see same data Availability Every request gets response Partition Tolerance Network splits OK CP System Consistent + Partition Tolerant Ex: HBase, MongoDB AP System Available + Partition Tolerant Ex: Cassandra, DynamoDB CA System Consistent + Available Ex: RDBMS (single node) Cassandra TUNABLE! Cassandra's Superpower: Tunable Consistency Choose CL=ALL (CP-like) or CL=ONE (AP-like) per query! Adjust trade-offs dynamically!

CAP Theorem Deep Dive

The Three Properties:

  • Consistency (C): All nodes see the same data at the same time - every read receives the most recent write
  • Availability (A): Every request receives a response (success or failure) - no request hangs forever
  • Partition Tolerance (P): System continues operating despite network partitions (splits between nodes)

Why You Can't Have All Three:

During a network partition (split brain), you must choose:

  • CP (Consistency + Partition): Reject requests to maintain consistency → Sacrifice Availability
  • AP (Availability + Partition): Accept requests despite split → Sacrifice Consistency (temporary)
  • CA (Consistency + Availability): Only works with no partitions → Not realistic for distributed systems

Cassandra's Choice:

Cassandra is fundamentally an AP system (Availability + Partition Tolerance), BUT with tunable consistency levels, you can achieve CP-like behavior when needed:

  • CL=ONE: Maximum availability (AP)
  • CL=QUORUM: Balanced (mostly consistent, highly available)
  • CL=ALL: Maximum consistency (CP-like, but sacrifices availability)

🔢 Quorum Mathematics

Understanding how quorum is calculated and why it guarantees consistency.

Quorum Formula

QUORUM = (RF / 2) + 1
(Majority of replicas, rounded down)

Examples:

  • RF=3: QUORUM = (3/2) + 1 = 1 + 1 = 2
  • RF=5: QUORUM = (5/2) + 1 = 2 + 1 = 3
  • RF=7: QUORUM = (7/2) + 1 = 3 + 1 = 4
  • RF=1: QUORUM = (1/2) + 1 = 0 + 1 = 1 (same as ALL)

Why QUORUM Works:

The magic of quorum is the guaranteed overlap:

RF=3 Example:
Write QUORUM = 2 replicas must confirm
Read QUORUM = 2 replicas must respond

Why guaranteed consistent:
Total replicas: 3 (Node A, B, C)
Write QUORUM: 2 nodes (e.g., A + B)
Read QUORUM: 2 nodes (e.g., B + C)

Overlap: Node B appears in BOTH!
Node B has the latest write → Read sees it!

Mathematical proof:
Write (2) + Read (2) = 4 > RF (3)
4 > 3 means guaranteed overlap! ✅

Strong Consistency Math

Rule: R + W > RF
(Read CL + Write CL > RF)

RF=3 Examples:
• QUORUM + QUORUM = 2 + 2 = 4 > 3 ✅
• ONE + ALL = 1 + 3 = 4 > 3 ✅
• TWO + TWO = 2 + 2 = 4 > 3 ✅

Counter-examples:
• ONE + ONE = 1 + 1 = 2 < 3 ❌
• ONE + TWO = 1 + 2 = 3 = 3 ❌
(Need > not ≥)

Failure Tolerance

Max Failures with QUORUM:
RF=3, QUORUM=2
Can lose: RF - QUORUM = 1 node

RF=5, QUORUM=3
Can lose: RF - QUORUM = 2 nodes

RF=7, QUORUM=4
Can lose: RF - QUORUM = 3 nodes

Pattern:
QUORUM tolerates (RF - 1) / 2 failures
(Always rounds down)

📊 All Consistency Levels Explained

Complete reference of every consistency level in Cassandra.

Level Replicas Speed Availability Use Case
ONE 1 / RF ⚡ Fastest 🟢 Highest Analytics, logs, metrics
TWO 2 / RF ⚡ Very Fast 🟢 High Rarely used (prefer QUORUM)
THREE 3 / RF ⚡ Fast 🟡 Medium Specific 3-replica scenarios
QUORUM (RF/2)+1 ⚡ Balanced 🟡 Good RECOMMENDED default!
ALL RF / RF 🐌 Slowest 🔴 Lowest Critical operations only
LOCAL_ONE 1 in local DC ⚡ Fastest 🟢 Highest Multi-DC, low latency reads
LOCAL_QUORUM Majority local ⚡ Fast 🟢 High MULTI-DC default!
EACH_QUORUM Quorum in ALL DCs 🐌 Slow 🔴 Low Critical multi-DC writes
SERIAL Quorum (LWT) 🐌 Slowest 🔴 Lowest Lightweight transactions
LOCAL_SERIAL Local quorum (LWT) 🐌 Very Slow 🔴 Low Multi-DC LWT

Recommended Defaults

Single Datacenter:

  • Write: QUORUM (balanced consistency + availability)
  • Read: QUORUM (strong consistency with QUORUM writes)
  • Fast Reads: ONE (if eventual consistency acceptable)

Multi-Datacenter:

  • Write: LOCAL_QUORUM (fast, DC-local majority)
  • Read: LOCAL_QUORUM (low latency, strong local consistency)
  • Critical Writes: EACH_QUORUM (all DCs must confirm)

⚙️ Tunable Consistency: The Power of Choice

Mix and match read and write consistency levels for perfect balance!

Tunable Consistency: Read + Write Combinations (RF=3) WRITE CL → READ CL ↓ ONE QUORUM ALL ONE QUORUM ALL R+W = 2 Eventual R+W = 3 Eventual R+W = 4 ✓ STRONG R+W = 3 Eventual R+W = 4 ✓ STRONG R+W = 5 ✓ STRONG R+W = 4 ✓ STRONG R+W = 5 ✓ STRONG R+W = 6 ✓ STRONG Common Patterns Fast Analytics (ONE + ONE) Fastest reads & writes Use: Logs, metrics, time-series Trade-off: Eventual consistency Balanced (QUORUM + QUORUM) Strong consistency + good availability Use: Most production workloads Trade-off: Moderate latency Fast Writes (ONE + ALL) Writes fast, reads check all replicas Use: Write-heavy, read-critical accuracy Trade-off: Slow reads Maximum Safety (ALL + ALL) Absolute consistency, all replicas always Use: Financial transactions, critical data Trade-off: Slowest, least available Strong Consistency Rule R + W > RF Read CL + Write CL > Replication Factor Examples (RF=3): QUORUM + QUORUM 2 + 2 = 4 > 3 ✓ ONE + ALL 1 + 3 = 4 > 3 ✓ ALL + ONE 3 + 1 = 4 > 3 ✓ TWO + TWO 2 + 2 = 4 > 3 ✓ NOT Strong (RF=3): ONE + ONE 1 + 1 = 2 < 3 ✗ ONE + TWO 1 + 2 = 3 = 3 ✗ TWO + ONE 2 + 1 = 3 = 3 ✗ Must be GREATER THAN, not equal!

Choosing the Right Combination

Decision Framework:

  • Need Strong Consistency? Use R + W > RF (e.g., QUORUM + QUORUM)
  • Write-Heavy Workload? Fast writes (ONE or LOCAL_ONE), slower reads (QUORUM or ALL)
  • Read-Heavy Workload? Fast reads (ONE or LOCAL_ONE), slower writes (QUORUM or ALL)
  • Maximum Availability? ONE + ONE (accepts eventual consistency)
  • Critical Data? ALL + ALL (sacrifices availability for certainty)

Production Patterns:

  • E-commerce Catalog: Write QUORUM, Read ONE (fast browsing, eventual consistency OK)
  • User Sessions: Write LOCAL_QUORUM, Read LOCAL_ONE (low latency, multi-DC)
  • Financial Ledger: Write QUORUM, Read QUORUM (strong consistency required)
  • Metrics/Logging: Write ONE, Read ONE (maximum throughput, eventual OK)
  • User Profile: Write ALL, Read ONE (ensure profile written everywhere, fast reads)

🖥️ Interactive Consistency Calculator

Experiment with different consistency levels and see the results!

Consistency Level Calculator
🖥️ Consistency Calculator Ready!
Select RF, Write CL, and Read CL, then click Calculate.

Try these combinations:
1. RF=3, Write=QUORUM, Read=QUORUM → Strong consistency
2. RF=3, Write=ONE, Read=ONE → Eventual consistency
3. RF=3, Write=ONE, Read=ALL → Strong consistency, fast writes
4. RF=5, Write=QUORUM, Read=QUORUM → Tolerates 2 failures!

✅ Best Practices & Common Mistakes

Production-grade recommendations for consistency configuration.

✅

Best Practices

  • Default to QUORUM: Best balance for most use cases
  • Strong Consistency: R + W > RF (e.g., QUORUM + QUORUM)
  • Multi-DC: Use LOCAL_QUORUM for low latency
  • Test Failures: Simulate node downs to verify behavior
  • Monitor Latency: Track p99 latencies by CL
  • Document Choices: Record why each table uses specific CL
  • Per-Query CL: Set CL per query, not globally
  • Client Timeouts: Set higher timeouts for ALL, lower for ONE
❌

Common Mistakes

  • Using ALL Everywhere: Destroys availability, very slow
  • Using ONE Everywhere: Weak consistency, data loss risk
  • R + W = RF: Not strong! Must be > RF
  • Ignoring Multi-DC: Use LOCAL_* for multi-DC deployments
  • No Testing: Untested CLs fail in production
  • ONE + ONE for Money: Never use weak CL for financial data
  • Mixing RF: Different RF per table complicates CL choices
  • No Monitoring: Can't optimize what you don't measure

Critical Warnings

Dangerous Consistency Patterns:

  • ONE + ONE for Critical Data: Can lose writes if node fails before replication! Use QUORUM minimum.
  • ALL in Production: ANY node failure = total unavailability. Only use for truly critical operations.
  • EACH_QUORUM for Everything: Extremely slow (cross-DC latency). Reserve for critical multi-DC writes.
  • R + W = RF: NOT strong consistency! Example: RF=3, ONE + TWO = 3 (not > 3). No overlap guarantee!

Real-World Failure Example:

Company X used Write ONE + Read ONE for user accounts
Node A receives write for user "alice"
Write succeeds on Node A only
Node A crashes 5 seconds later (before replication)
Client reads from Node B (doesn't have data yet)
Result: User "alice" account disappeared! 💥

Fix: Changed to Write QUORUM + Read QUORUM
Now 2/3 nodes must confirm write (redundancy!)

💼 Interview Questions & Answers

Master these 25 essential questions on Cassandra Consistency & Quorum - from beginner to expert level!

1 What is QUORUM and how is it calculated? ▼

Answer:

QUORUM = (RF / 2) + 1 (rounded down)

Definition: The majority of replicas - more than half must respond for operation to succeed.

Examples:

  • RF=1: QUORUM = (1/2) + 1 = 0 + 1 = 1 (same as ALL)
  • RF=3: QUORUM = (3/2) + 1 = 1 + 1 = 2 (majority)
  • RF=5: QUORUM = (5/2) + 1 = 2 + 1 = 3 (majority)
  • RF=7: QUORUM = (7/2) + 1 = 3 + 1 = 4 (majority)

Why QUORUM Works:

  • Guaranteed Overlap: Write QUORUM + Read QUORUM always overlap on at least one node
  • Strong Consistency: QUORUM + QUORUM provides linearizable reads
  • Fault Tolerance: Can tolerate (RF - QUORUM) failures
  • Example (RF=3): Write to 2 nodes, read from 2 nodes → must overlap on at least 1 node with latest data
2 What's the formula for strong consistency? ▼

Answer:

R + W > RF (Read CL + Write CL must be GREATER THAN Replication Factor)

Examples for RF=3:

  • QUORUM + QUORUM: 2 + 2 = 4 > 3 ✓ (Strong)
  • ONE + ALL: 1 + 3 = 4 > 3 ✓ (Strong)
  • ALL + ONE: 3 + 1 = 4 > 3 ✓ (Strong)
  • TWO + TWO: 2 + 2 = 4 > 3 ✓ (Strong)
  • ONE + ONE: 1 + 1 = 2 < 3 ✗ (Eventual)
  • ONE + TWO: 1 + 2 = 3 = 3 ✗ (NOT strong - must be >)

Why This Works:

When R + W > RF, the read and write sets MUST overlap - at least one node will have the latest write. This guarantees you always read the most recent data.

3 What's the difference between QUORUM and LOCAL_QUORUM? ▼

Answer:

QUORUM: Majority of ALL replicas across ALL datacenters

  • RF=3 (single DC): 2 nodes
  • RF=6 (2 DCs, 3 per DC): 4 nodes total (could be 2 from each DC, or 3+1, or 4+0)
  • Waits for cross-DC responses → high latency

LOCAL_QUORUM: Majority of replicas in LOCAL datacenter only

  • RF=3 per DC: 2 nodes in local DC
  • Ignores remote datacenters → low latency
  • Most common for multi-DC deployments

Example (2 DCs, RF=3 per DC):

  • QUORUM: Need 4/6 nodes (could wait for remote DC)
  • LOCAL_QUORUM: Need 2/3 nodes in local DC only (fast!)

When to Use:

  • QUORUM: Single DC deployments
  • LOCAL_QUORUM: Multi-DC deployments (recommended default)
4 Explain the CAP theorem in relation to Cassandra. ▼

Answer:

CAP Theorem: In a distributed system during a network partition, you can only have 2 of 3: Consistency, Availability, Partition Tolerance.

Cassandra's Choice:

Cassandra is fundamentally an AP system (Availability + Partition Tolerance):

  • Availability: Accepts reads/writes even during partitions
  • Partition Tolerance: Continues operating despite network splits
  • Consistency: Sacrificed by default (eventual consistency)

BUT Cassandra is Tunable!

  • CL=ONE: Maximum AP (highest availability, eventual consistency)
  • CL=QUORUM: Balanced (good availability, strong consistency)
  • CL=ALL: CP-like (maximum consistency, low availability)

Key Point: Cassandra lets you choose your position on CAP spectrum PER QUERY!

5 What happens when you use Write ONE and a node fails immediately after? ▼

Answer:

Potential Data Loss Scenario!

What Happens:

  1. Write ONE: Coordinator writes to Node A only (1/3 replicas)
  2. Write succeeds immediately (client gets OK)
  3. Node A crashes before replicating to Nodes B and C
  4. Data is LOST! (only copy was on Node A)

Mitigations:

  • Hinted Handoff: Coordinator stores hint for failed replicas, BUT only helps if coordinator doesn't also fail
  • Read Repair: Won't help - no other node has the data to repair from
  • Anti-Entropy Repair: Can't repair data that doesn't exist anywhere

Solution:

Use Write QUORUM (2/3 nodes) minimum for important data. This ensures at least 2 copies exist, so losing 1 node doesn't lose data.

6 What is EACH_QUORUM and when should you use it? ▼

Answer: EACH_QUORUM requires QUORUM acknowledgments from EVERY datacenter. Example: 2 DCs with RF=3 each → need 2/3 nodes from DC1 AND 2/3 from DC2 = 4 total confirmations. Use for critical writes that must be immediately available in all DCs (e.g., user authentication). WARNING: Very slow (cross-DC latency), low availability (any DC down = write fails). Only use for truly critical multi-DC operations.

7 Can you have strong consistency with RF=1? ▼

Answer: Yes, but with ZERO fault tolerance! RF=1 means only 1 copy of data exists. QUORUM = (1/2) + 1 = 1, ALL = 1, ONE = 1 (all the same). Strong consistency trivially achieved (R + W > RF: 1 + 1 = 2 > 1), BUT if that single node fails, data is permanently lost. NEVER use RF=1 in production - absolute minimum is RF=3 for any important data.

8 Why is Write QUORUM + Read ONE not strongly consistent? ▼

Answer: R + W = 1 + 2 = 3, which equals RF (not greater than). No guaranteed overlap! Example: RF=3, write goes to Nodes A+B (QUORUM=2), read from Node C (ONE=1). Node C might not have latest write yet. Need R + W > RF for guaranteed overlap, so 1 + 2 = 3 is NOT > 3. Use QUORUM + QUORUM (2 + 2 = 4 > 3) for strong consistency.

9 How many node failures can QUORUM tolerate? ▼

Answer: QUORUM can tolerate (RF - QUORUM) node failures. Examples: RF=3, QUORUM=2 → can lose 1 node (3-2=1). RF=5, QUORUM=3 → can lose 2 nodes (5-3=2). RF=7, QUORUM=4 → can lose 3 nodes (7-4=3). General pattern: QUORUM tolerates (RF-1)/2 failures (rounded down). This is why odd RF values are recommended (RF=3, 5, 7) - maximize fault tolerance per replica.

10 What is the difference between consistency level and replication factor? ▼

Answer: Replication Factor (RF): How many copies of data exist (set per keyspace, e.g., RF=3 = 3 copies). Consistency Level (CL): How many replicas must respond for operation success (set per query, e.g., CL=QUORUM = 2/3 must respond). RF is about redundancy/durability, CL is about consistency/latency trade-off. Example: RF=3 means 3 nodes have data, but CL=ONE means only 1 node needs to respond for read to succeed.

11 Can you change consistency level per query? ▼

Answer: YES! This is Cassandra's "tunable consistency" superpower. Every single query can specify its own CL. Example: Fast analytics query uses CL=ONE, critical transaction uses CL=QUORUM, account deletion uses CL=ALL - all in the same application. In CQL: SELECT * FROM users USING CONSISTENCY QUORUM; or in drivers: session.execute(query, consistency_level=ConsistencyLevel.QUORUM). This per-query tunability is what makes Cassandra unique vs traditional databases.

12 What happens during a network partition with CL=ALL? ▼

Answer: Complete unavailability! CL=ALL requires ALL replicas to respond. During partition: Client in Partition A cannot reach nodes in Partition B → timeout → query fails. This is why ALL is dangerous in production. Example: RF=3, nodes split 2-1 across partition. Client with CL=ALL cannot complete ANY query (needs all 3, can only reach 2). Even reads fail! Use ALL only for truly critical operations where you can tolerate downtime. For most cases, QUORUM provides strong consistency with better availability.

13 How does timestamp tie-breaking work with eventual consistency? ▼

Answer: Cassandra uses Last Write Wins (LWW) with microsecond timestamps. When replicas have different values: (1) Compare timestamps, (2) Newest timestamp wins, (3) If timestamps equal, compare values lexicographically (larger value wins). Example: Node A has value "X" (timestamp 1000), Node B has "Y" (timestamp 2000). Read QUORUM gets both → returns "Y" (newer). This is why client-side timestamps and NTP synchronization are critical! Skewed clocks can cause newer writes to be overwritten by older writes.

14 What is SERIAL consistency level? ▼

Answer: SERIAL is for Lightweight Transactions (LWT) using Paxos consensus algorithm. Provides linearizable consistency (compare-and-set operations). Example: UPDATE users SET balance = 100 IF balance = 50; uses SERIAL to ensure atomic check-and-update. SERIAL requires QUORUM participation in 4-phase Paxos (prepare, promise, propose, accept). Extremely slow (4 round trips!) but guarantees no concurrent modification. Use sparingly - only for critical operations like account balance updates, seat reservations, distributed locks. Most queries should use regular QUORUM.

15 Why use Write ONE + Read ALL instead of QUORUM + QUORUM? ▼

Answer: Optimize for write-heavy workloads with critical read accuracy. Write ONE: Fast writes (1 node confirms), Read ALL: Absolutely certain reads (all nodes checked). Both provide strong consistency (1 + 3 = 4 > 3 for RF=3). Use case: Logging system where writes are frequent (millions/sec) but reads are rare and must be 100% accurate. Trade-off: Writes blazing fast, reads very slow and unavailable if any node down. QUORUM+QUORUM is usually better balanced choice.

16 Does higher consistency level make data more durable? ▼

Answer: NO! Consistency level affects READ/WRITE acknowledgment, not durability. With RF=3, data is written to 3 nodes REGARDLESS of CL. Difference: CL=ONE: Client gets OK after 1 write completes (but all 3 still happen via hinted handoff). CL=ALL: Client waits for all 3 writes to confirm. Durability comes from RF (replication factor), not CL. Even with CL=ONE, you still have 3 copies (eventually). Higher CL just means you WAIT for confirmation, ensuring immediate visibility across replicas.

17 How do you achieve strong consistency in multi-DC Cassandra? ▼

Answer: Use EACH_QUORUM for writes (requires QUORUM in every DC), LOCAL_QUORUM for reads. Example: 2 DCs, RF=3 per DC. EACH_QUORUM write: Waits for 2/3 nodes in DC1 AND 2/3 in DC2 (4 confirmations total). LOCAL_QUORUM read: Checks 2/3 nodes in local DC only. This ensures writes are immediately durable in all DCs, reads are fast and locally consistent. WARNING: EACH_QUORUM very slow (cross-DC latency), fails if any DC down. Alternative: LOCAL_QUORUM writes with periodic repair for eventual consistency across DCs.

18 What's the latency difference between ONE, QUORUM, and ALL? ▼

Answer: Latency = time to wait for slowest required response. ONE: Returns after fastest node (typically 1-5ms local). QUORUM: Returns after second-fastest of 3 nodes (typically 5-15ms, waiting for 2nd response). ALL: Returns after slowest of 3 nodes (could be 50-100ms if one node slow). Example p99 latencies: ONE: 3ms, QUORUM: 12ms, ALL: 85ms. Multi-DC amplifies difference: LOCAL_ONE: 3ms, LOCAL_QUORUM: 12ms, QUORUM (cross-DC): 150ms, ALL (cross-DC): 300ms+. Choose based on latency budget!

19 Can you mix different consistency levels for reads and writes on the same table? ▼

Answer: YES! This is the essence of tunable consistency. Common patterns: Write QUORUM, Read ONE: Writes are durable, reads are fast (eventual consistency). Write ONE, Read QUORUM: Writes are fast, reads ensure recent data. Write QUORUM, Read QUORUM: Strong consistency everywhere. Same table can have different CLs per operation type. Example: E-commerce catalog might use Write QUORUM (ensure inventory updates durable) + Read ONE (fast browsing), but checkout uses Read QUORUM + Write QUORUM (ensure accurate inventory for purchase).

20 What happens if you use Read QUORUM but only 1 node is available? ▼

Answer: Query fails with Unavailable Exception! Read QUORUM (RF=3) requires 2/3 nodes. Only 1 available → cannot meet QUORUM → coordinator returns error to client immediately (no timeout). This is by design - Cassandra won't return stale data if it can't guarantee consistency level. Client options: (1) Retry with lower CL like ONE (if acceptable), (2) Wait for nodes to recover, (3) Route to different DC. This is why QUORUM requires (RF/2)+1 nodes UP. If you need availability during failures, use CL=ONE.

21 How does read repair interact with consistency levels? ▼

Answer: Read repair happens AFTER CL is satisfied. Flow: (1) Coordinator requests from CL replicas (e.g., QUORUM=2), (2) Returns fastest valid response to client, (3) Async checks digest from remaining replicas, (4) If mismatch detected, triggers read repair to sync all replicas. Higher CL = more replicas checked = faster inconsistency detection. CL=ONE: Only 1 replica checked, inconsistencies may persist. CL=QUORUM: 2 checked, likely catches inconsistencies. CL=ALL: All checked, guaranteed repair. Read repair is optimization, not replacement for periodic nodetool repair!

22 Why would you ever use TWO or THREE instead of QUORUM? ▼

Answer: Rarely! TWO/THREE are mostly legacy or very specific use cases. Possible reason: Explicit control over exact replica count (e.g., THREE ensures 3 specific nodes confirmed, QUORUM is formula-based). But QUORUM is almost always better: (1) Scales with RF changes (RF=3→5, QUORUM auto-adjusts 2→3), (2) Mathematical guarantee for strong consistency, (3) Industry standard terminology. Exception: RF=4, QUORUM=3, but you want exactly 2 for specific reason - use TWO. Recommendation: Stick with ONE/QUORUM/ALL for 99% of use cases.

23 How do you debug consistency level issues in production? ▼

Answer: Tools & Techniques: (1) Enable query tracing: TRACING ON; SELECT ...; shows which nodes responded, (2) Check coordinator logs for Unavailable/Timeout exceptions, (3) Monitor metrics: ConsistencyLevel breakdown, latency histograms per CL, (4) Use nodetool status to verify node states, (5) Check nodetool tablestats for replica counts, (6) Temporarily lower CL to isolate availability vs consistency issue, (7) Run nodetool repair to fix inconsistencies. Common issues: Nodes actually down, Network partition, Clock skew causing timestamp conflicts, Application using wrong CL.

24 What's the impact of increasing RF from 3 to 5 on consistency? ▼

Answer: QUORUM changes from 2 to 3 nodes! RF=3: QUORUM=2, tolerate 1 failure. RF=5: QUORUM=3, tolerate 2 failures. Impacts: (1) Availability: Better (can lose 2 nodes vs 1), (2) Latency: Slightly worse (wait for 3rd response instead of 2nd), (3) Storage: 67% more disk space (5 copies vs 3), (4) Write throughput: Same (still parallel writes), (5) Network: More bandwidth (5 replicas). Strong consistency formula still works: R + W > RF (3 + 3 = 6 > 5 with QUORUM+QUORUM). Trade-off: Better fault tolerance, higher storage cost.

25 Design a consistency strategy for a global social media app. ▼

Answer:

Architecture: 3 datacenters (US, EU, ASIA), RF=3 per DC (9 total copies)

User Posts (write-heavy, eventual OK):

  • Write: LOCAL_QUORUM (2/3 in local DC, ~10ms)
  • Read: LOCAL_ONE (fastest local node, ~3ms)
  • Reason: Users can tolerate slight delay seeing others' posts (eventual consistency), prioritize write speed

User Profile (read-heavy, need accuracy):

  • Write: EACH_QUORUM (durable in all DCs)
  • Read: LOCAL_QUORUM (accurate local reads)
  • Reason: Profile changes rare but must be globally consistent

Private Messages (critical accuracy):

  • Write: LOCAL_QUORUM (balance speed + safety)
  • Read: LOCAL_QUORUM (ensure message not lost)
  • Reason: Strong consistency locally, async replication to other DCs

Analytics/Metrics (best effort):

  • Write: ONE (maximum throughput)
  • Read: ONE (eventual consistency fine)
  • Reason: Millions of metrics/sec, approximate counts acceptable

🎓 Chapter Summary

You now master Cassandra's Tunable Consistency!

Key Formulas:

  • QUORUM = (RF / 2) + 1 (majority of replicas)
  • Strong Consistency: R + W > RF (guaranteed overlap)
  • Fault Tolerance: RF - QUORUM (max failures)

CAP Theorem:

  • Cassandra is AP (Availability + Partition Tolerance)
  • But TUNABLE per query: CL=ONE (max AP) to CL=ALL (CP-like)
  • QUORUM provides balanced consistency + availability

Production Recommendations:

  • Single DC: Write QUORUM, Read QUORUM (default)
  • Multi-DC: Write LOCAL_QUORUM, Read LOCAL_QUORUM
  • Critical Ops: EACH_QUORUM or ALL (sparingly!)
  • Fast Analytics: Write ONE, Read ONE (eventual OK)
  • Always test failure scenarios before production!
Advertisement

Responsive Ad