LOCAL - Multi-Datacenter Mastery

Complete Answer:

LOCAL_QUORUM is a consistency level that requires QUORUM (majority) of replicas to acknowledge within the local datacenter only, not globally. It's the recommended default for multi-DC Cassandra deployments.

How it works:

• Write: Coordinator sends write to all replicas globally but only waits for QUORUM of local DC replicas to ACK before returning success
• Example: RF=3 per DC → Wait for 2 ACKs from local DC only (~8ms), remote DCs updated asynchronously (~150ms)
• Read: Contact QUORUM of replicas in local DC only

Why essential for multi-DC:

• Cross-DC latency problem: US→Europe = 150ms, US→Asia = 250-300ms (physics - can't avoid!)
• With regular QUORUM: Need 2 ACKs from ANY DCs = might wait for Asia = 300ms (disaster!)
• With LOCAL_QUORUM: Need 2 ACKs from local DC only = ~8ms (18-35x faster!)
• Strong consistency locally: R_local + W_local > RF_local = 2 + 2 = 4 > 3 (guaranteed fresh reads locally)
• Eventual across DCs: Cross-DC replication happens async (100-200ms) - acceptable for most use cases

Real production impact:

• Uber: Ride matching ~10ms with LOCAL_QUORUM vs 200-300ms with cross-DC QUORUM
• Netflix: Account updates ~8ms locally, billions saved in infrastructure
• Apple: iCloud sync instant within region, eventual globally

Key insight: In multi-DC deployments, LOCAL_QUORUM gives you the best of both worlds: strong consistency where you need it (local DC) and performance that doesn't suffer from cross-continent network latency. This is why virtually every major company with global Cassandra deployments uses LOCAL_QUORUM as their default consistency level!

Complete Answer:

Both ONE and LOCAL_ONE wait for only 1 replica to acknowledge, but they differ crucially in which replica that can be.

Key Difference:

• ONE: Waits for 1 replica from ANY datacenter globally (could be local or remote!)
• LOCAL_ONE: Waits for 1 replica from local datacenter only (guaranteed local)

The Problem with ONE in Multi-DC:

• Scenario: You're a US client reading with CL=ONE from a cluster with DCs in US, Europe, and Asia
• Problem: Cassandra might randomly pick a replica in Asia datacenter!
• Result: Read latency = 250-300ms (cross-Pacific network) even though there's a replica right next to you in US
• Impact: Unpredictable latency - sometimes 3ms, sometimes 300ms!

Why LOCAL_ONE exists:

• Guarantee: Always reads from local DC = predictable low latency (~3-5ms)
• Use case: High-volume non-critical data in multi-DC (analytics, logs, metrics)
• Benefit: Fastest possible local reads, no cross-DC network hops
• Trade-off: Eventual consistency both locally AND across DCs

Detailed Example:

Setup: 3 DCs (US, EU, Asia), RF=3 per DC, US client

• With ONE:
  • Cassandra picks random replica globally
  • 33% chance: US replica = 3ms ✓
  • 33% chance: EU replica = 150ms ❌
  • 33% chance: Asia replica = 300ms ❌
  • P50 latency: ~150ms (unpredictable!)
• With LOCAL_ONE:
  • Always picks US replica (local DC)
  • 100% chance: US replica = 3-5ms ✓
  • P50 latency: 3ms (predictable!)

When to use each:

• Use LOCAL_ONE when: Multi-DC deployment + high-volume non-critical data (analytics, logs)
• Use ONE when: Single DC deployment (LOCAL_ONE = ONE in single DC)
• Never use ONE in multi-DC: Unpredictable cross-DC latency!

Production recommendation: If you have multiple datacenters, always use LOCAL_ variants (LOCAL_ONE, LOCAL_QUORUM) instead of ONE or QUORUM. This ensures predictable latency and avoids unexpected cross-datacenter network hops!

Complete Answer:

When a datacenter fails with LOCAL_QUORUM, writes/reads to that specific datacenter will fail, but other datacenters continue operating normally. This is an important trade-off to understand.

What Happens During DC Failure:

• Failed DC: Clients trying to write/read with LOCAL_QUORUM to the failed DC will get errors (can't get QUORUM from a down DC)
• Other DCs: Continue working normally - clients routed to healthy DCs operate fine
• Example: US DC fails → US clients get errors, EU/Asia clients work normally

Contrast with regular QUORUM:

• With QUORUM: If US DC fails, can still get QUORUM from EU + Asia DCs = writes succeed
• Better availability during DC failures
• BUT: Normal operation = 150-300ms latency (unacceptable!)
• Trade-off: 99.9% of time speed vs 0.1% of time availability

How to Handle DC Failures with LOCAL_QUORUM:

1. Client-Side Failover (Most Common):

• Setup: DC-aware load balancer (e.g., AWS Route 53, Consul)
• Normal operation: US clients → US DC, EU clients → EU DC
• On US DC failure: Load balancer detects failure, routes US clients to EU DC
• Result: US clients now write to EU DC with LOCAL_QUORUM = works! (slightly higher latency ~150ms but better than complete outage)
• Recovery: When US DC comes back, clients gradually fail back

2. Application-Level Retry:

// Example retry logic
try {
    session.execute(query, LOCAL_QUORUM); // Try local DC
} catch (NoHostAvailableException e) {
    // Local DC down, retry with QUORUM to fail over to other DCs
    session.execute(query, QUORUM);
}

3. Use EACH_QUORUM for Critical Writes:

• For critical data: Use EACH_QUORUM instead of LOCAL_QUORUM
• EACH_QUORUM: Requires QUORUM in EACH datacenter
• Benefit: If one DC fails, other DCs still have quorum of data
• Cost: Slower (cross-DC latency) but better availability
• Use case: Billing, payment processing, critical business data

4. Hinted Handoff for Recovery:

• During outage: Cassandra stores "hints" for failed DC
• When DC recovers: Hints are replayed to catch up missed writes
• Timeline: Usually catches up in minutes to hours
• Limitation: Hints only stored for 3 hours by default (configurable)

5. Monitor DC Health:

• Metrics: Monitor DC availability, latency, error rates
• Alerts: Immediate notification on DC failures
• Tools: DataStax OpsCenter, Prometheus + Grafana
• SLOs: Set DC availability SLOs (e.g., 99.95%)

Production Decision Framework:

• Most companies choose: LOCAL_QUORUM + client failover
• Reasoning: 99.9% of time = 10ms latency, 0.1% of time (DC failure) = 150ms + failover
• Alternative: Always using QUORUM = 100% of time = 150ms (unacceptable for user experience)
• Critical data: Use EACH_QUORUM for must-never-lose data (billing, compliance)

Real-world example: When AWS US-EAST-1 had a major outage, companies with LOCAL_QUORUM + proper failover automatically routed traffic to US-WEST and EU datacenters. Users experienced slightly higher latency but service stayed up. Companies without failover had complete outages in affected regions.

Complete Answer:

With LOCAL_QUORUM, writes are acknowledged locally fast, then replicated to remote datacenters asynchronously in the background. Understanding this timeline is crucial for multi-DC deployments.

Complete Write Timeline (US client, 3 DCs: US, EU, Asia):

T=0ms:

• Client in US sends write with CL=LOCAL_QUORUM
• Coordinator node in US receives write
• Coordinator determines which replicas need the data (3 in US, 3 in EU, 3 in Asia = 9 total)

T=0-8ms (Local DC writes):

• Coordinator sends write to all 3 US replicas in parallel
• T=5ms: US Replica 1 ACKs ✓
• T=8ms: US Replica 2 ACKs ✓
• LOCAL_QUORUM satisfied! (2 of 3 local replicas)
• Coordinator returns SUCCESS to client
• Client latency: 8ms (user sees instant confirmation)

T=8-150ms (Cross-DC replication - Async):

• Coordinator already sent write to EU/Asia coordinators at T=0
• EU coordinator writes to 3 EU replicas (happens in parallel with US writes)
• Asia coordinator writes to 3 Asia replicas (happens in parallel)
• T=150ms: EU replicas have data ✓ (US→EU network = ~120-150ms)
• T=280ms: Asia replicas have data ✓ (US→Asia network = ~250-300ms)
• Client doesn't wait for EU/Asia - already got success at T=8ms!

T=280ms+: Global Consistency Achieved:

• All 9 replicas globally now have the data
• System is globally consistent (eventual consistency complete)
• Total replication time: ~280ms
• But client latency was only 8ms!

What Happens During Async Replication:

• Coordinator role: US coordinator sends write to EU/Asia coordinators immediately (T=0)
• Remote coordinators: Each DC's coordinator writes to its local replicas independently
• No blocking: US client doesn't wait for EU/Asia coordinators to finish
• Parallel writes: All DCs receive and process write simultaneously
• Network latency: Cross-DC network adds 150-300ms but doesn't block client

Reading During Replication Window:

Scenario: US client writes at T=0, EU client reads at T=50ms

• Problem: EU DC hasn't received data yet (arrives at T=150ms)
• EU read with LOCAL_QUORUM: Returns old data (stale!) ❌
• This is expected! Eventual consistency across DCs
• By T=150ms: EU reads return new data ✓

Replication Mechanics:

• Protocol: Cassandra's inter-DC streaming
• Optimization: Batching (multiple writes bundled together)
• Compression: Cross-DC traffic is compressed
• Retry logic: Automatic retry if cross-DC write fails
• Hinted handoff: If remote DC temporarily down, hints stored and replayed later

Monitoring Cross-DC Replication:

-- Check cross-DC replication lag
nodetool netstats

-- Example output:
Mode: NORMAL
Datacenter: US
  Outbound: 0 bytes
  Inbound: 12.5 MB
Datacenter: EU  
  Outbound: 8.2 MB (lag: 120ms)
  Inbound: 0 bytes
Datacenter: Asia
  Outbound: 6.1 MB (lag: 280ms)
  Inbound: 0 bytes

Typical Replication Lags:

• Same region: US-East → US-West = 60-80ms
• Cross continent: US → Europe = 120-150ms
• Across Pacific: US → Asia = 250-300ms
• Around globe: US → Australia = 180-220ms

Key insight: LOCAL_QUORUM achieves strong consistency locally (8ms) while eventual consistency globally (150-300ms). The client never waits for cross-DC replication, which is why it's 15-35x faster than regular QUORUM. This async replication model is what enables global-scale applications to maintain both low latency AND geographic redundancy!

Complete Answer:

These three consistency levels represent different trade-offs between latency, consistency strength, and cross-DC behavior in multi-datacenter deployments.

Detailed Comparison (3 DCs: US, EU, Asia, RF=3 per DC):

LOCAL_QUORUM (Recommended Default):

• Behavior: Requires QUORUM (2 of 3) from local DC only
• Write latency: 8-12ms (local QUORUM time)
• Read latency: 8-12ms (local QUORUM time)
• Throughput: 30,000+ writes/sec per node
• Consistency: Strong locally, eventual across DCs
• DC failure impact: Writes to failed DC fail, other DCs work
• Use case: 99% of multi-DC production workloads!
• Examples: User profiles, orders, account settings, most business data

QUORUM (Dangerous in Multi-DC):

• Behavior: Requires QUORUM from ANY replicas globally (with RF=3 per DC, total RF=9, needs 5 ACKs)
• Write latency: 150-300ms (must wait for cross-DC ACKs)
• Read latency: 150-300ms (might query remote DCs)
• Throughput: 2,000 writes/sec (15x less than LOCAL_QUORUM!)
• Consistency: Strong globally
• DC failure impact: Can survive 1 DC failure (get QUORUM from remaining DCs)
• Problem: Unacceptable latency for user-facing operations
• Use case: Never use in multi-DC! (Use EACH_QUORUM if you need global consistency)

EACH_QUORUM (Critical Cross-DC):

• Behavior: Requires QUORUM in EACH datacenter (2 of 3 in US AND 2 of 3 in EU AND 2 of 3 in Asia)
• Write latency: 280-300ms (must wait for slowest DC)
• Read latency: 8-12ms (can read from local DC only)
• Throughput: 1,000 writes/sec (30x less than LOCAL_QUORUM!)
• Consistency: Strong in ALL datacenters
• DC failure impact: If ANY DC fails, all writes fail (poor availability!)
• Use case: Critical business data that MUST be consistent globally
• Examples: Billing records, payment processing, audit logs, compliance data

Performance Comparison Table:

When to Choose Each:

Choose LOCAL_QUORUM (99% of cases):

• User-facing data: profiles, accounts, preferences
• Business operations: orders, inventory, reservations
• Application state: sessions, carts, user actions
• Why: Best balance of strong consistency + low latency
• Trade-off: Cross-DC eventual (acceptable for most use cases)

Choose EACH_QUORUM (Rare, <1% of cases):

• Financial data: billing records, payment transactions
• Compliance: audit logs, regulatory data
• Critical business: must-never-lose data
• Why: Need guarantee that ALL DCs have data before acknowledging
• Trade-off: Much slower, but worth it for critical data

NEVER Choose QUORUM in Multi-DC:

• It's a trap! Gives you cross-DC latency without cross-DC guarantees
• Worse than LOCAL_QUORUM (slower) without benefits of EACH_QUORUM (global consistency)
• If you need local consistency: use LOCAL_QUORUM
• If you need global consistency: use EACH_QUORUM
• There's no scenario where QUORUM makes sense in multi-DC!

Production Pattern (Hybrid Approach):

-- 99% of tables: LOCAL_QUORUM
CREATE TABLE users (...) WITH ...;
-- Use LOCAL_QUORUM for reads/writes

-- <1% of tables: EACH_QUORUM
CREATE TABLE billing_transactions (...) WITH ...;
-- Use EACH_QUORUM for writes (ensure all DCs have it)
-- Use LOCAL_QUORUM for reads (fast local reads)

Real-world example: Netflix uses LOCAL_QUORUM for 99% of their data (user accounts, viewing history, preferences) because the performance benefit is massive and cross-DC eventual consistency is fine. They use EACH_QUORUM only for critical billing data where they need absolute guarantee that payment records exist in all datacenters before acknowledging the transaction.

Key takeaway: In multi-datacenter deployments, LOCAL_QUORUM should be your default choice. Only deviate to EACH_QUORUM for truly critical data where you need guaranteed global consistency despite the huge performance cost. Never use plain QUORUM in multi-DC - it's almost always the wrong choice!