Multi-Datacenter Deployments

Answer:

LOCAL_QUORUM:

• Scope: Only nodes in the local datacenter
• Calculation: (Local RF / 2) + 1
• Example: RF=3 in US-East → LOCAL_QUORUM = 2 nodes in US-East
• Latency: ~5-10ms (local network only)
• Cross-DC: Data replicates asynchronously (client doesn't wait)

QUORUM (Global):

• Scope: All replicas across all datacenters
• Calculation: (Total RF / 2) + 1
• Example: RF=3 US-East + RF=3 US-West + RF=2 EU = 8 total → QUORUM = 5
• Latency: ~100-150ms (may need cross-DC queries)
• Cross-DC: Must wait for nodes in other datacenters

Detailed Example:

Configuration:
- US-East: RF=3
- US-West: RF=3
- EU-West: RF=2
- Total: 8 replicas

Write from US-East with LOCAL_QUORUM:
1. Client writes to coordinator in US-East
2. Coordinator writes to 3 nodes in US-East
3. Waits for 2 acks (LOCAL_QUORUM = 2 of 3)
4. Returns success to client (~5-10ms)
5. Data asynchronously replicates to US-West and EU-West

Write from US-East with QUORUM:
1. Client writes to coordinator in US-East
2. Coordinator writes to nodes in ALL datacenters
3. Waits for 5 acks total (could be from any DCs)
4. Might need: 2 from US-East + 2 from US-West + 1 from EU
5. Must wait for cross-DC network (~100-150ms)
6. Returns success to client

Why LOCAL_QUORUM is Recommended:

• Fast: 20-30x lower latency (5ms vs 100ms)
• Available: Doesn't depend on other DCs being up
• Consistent: Still provides strong consistency within DC
• Scalable: Each DC operates independently

When to Use QUORUM:

• Need global strong consistency immediately (rare)
• Reading data that might be in different DCs
• Very specific edge cases (1% of operations)

Answer:

Replication Type: Asynchronous

Cross-datacenter replication in Cassandra is asynchronous by default (when using LOCAL_QUORUM). Here's the complete flow:

Write Flow (LOCAL_QUORUM):

T=0ms: Client writes to coordinator in US-East
T=1ms: Coordinator identifies all 8 replica nodes:
       - 3 in US-East (N1, N2, N3)
       - 3 in US-West (N4, N5, N6)
       - 2 in EU-West (N7, N8)

T=2ms: Coordinator writes to LOCAL nodes (US-East):
       - N1: Write starts
       - N2: Write starts
       - N3: Write starts

T=5ms: LOCAL_QUORUM achieved:
       - N1: ✓ Ack received
       - N2: ✓ Ack received
       - N3: Still writing (async)

T=5ms: SUCCESS returned to client!
       ← Client doesn't wait for other DCs!

T=50-80ms: US-West receives data:
       - N4, N5, N6 write asynchronously
       - Client already got success

T=100-150ms: EU-West receives data:
       - N7, N8 write asynchronously
       - Client already got success

Synchronous Option (EACH_QUORUM):

If you need synchronous cross-DC replication, use EACH_QUORUM:

Write with EACH_QUORUM:
T=0ms: Client writes
T=5ms: US-East QUORUM achieved (2 of 3)
T=80ms: US-West QUORUM achieved (2 of 3)
T=150ms: EU-West QUORUM achieved (2 of 2 - both!)
T=150ms: SUCCESS returned to client

Client waits for QUORUM in EVERY datacenter!
Latency: ~150-200ms (slowest DC determines speed)

How Async Replication Works:

• Coordinator's Job: Coordinator node knows all replica locations via token ring
• Local First: Writes to local DC nodes immediately
• Remote Second: Sends writes to remote DC coordinators
• Remote Coordinators: Each DC has coordinators that handle local writes
• No Waiting: Client gets success after local writes complete

What if Remote DC is Down?

Scenario: Write with LOCAL_QUORUM, but EU-West is DOWN

T=0ms: Client writes
T=5ms: US-East QUORUM achieved → SUCCESS!
T=50ms: US-West receives data (async)
T=150ms: EU-West unreachable!

Solution: Hinted Handoff
- Coordinator stores "hint" locally
- Hint says: "Send this to EU-West when it's back"
- EU-West comes back online
- Coordinator replays all hints
- EU-West catches up automatically!

Hints stored for: 3 hours by default (configurable)

Consistency Guarantees:

• LOCAL_QUORUM: Strongly consistent within DC, eventually consistent globally
• EACH_QUORUM: Strongly consistent globally (but slower)
• Anti-Entropy: Background repair ensures eventual consistency even without hints

Real-World Usage:

• Netflix: 99% LOCAL_QUORUM, <1% EACH_QUORUM
• Uber: All LOCAL_QUORUM except financial transactions
• Apple: LOCAL_QUORUM for user data, EACH_QUORUM for billing

Answer:

Scenario: Complete datacenter failure (power, network, AWS region outage, etc.)

Immediate Effects:

Setup:
- US-East: RF=3 (3 nodes)
- US-West: RF=3 (3 nodes)
- EU-West: RF=2 (2 nodes)
- Total: 8 replicas globally

US-East FAILS (complete region outage):

Immediate (T=0):
✓ US-West: Still has ALL data (3 copies)
✓ EU-West: Still has ALL data (2 copies)
✓ Service continues uninterrupted
✓ Zero data loss

Impact on Users:
⚠️ US-East users: Routed to US-West
   - Latency increases: 5ms → 85ms
   - GeoDNS automatically redirects
   - Service still works, just slower

⚠️ US-West users: No change
   - Still query local DC
   - Latency unchanged: ~5-10ms

⚠️ EU users: No change
   - Still query local DC
   - Latency unchanged: ~5-10ms

What Cassandra Does Automatically:

• Gossip Protocol: Detects US-East nodes are down within seconds
• Client Drivers: Automatically stop sending queries to US-East
• Load Balancer: Routes US-East traffic to US-West
• Hinted Handoff: US-West and EU-West store hints for US-East
• No Manual Intervention: Everything automatic!

Consistency Level Behavior:

LOCAL_QUORUM (Recommended):
✓ Writes from US-East users → US-West:
  - US-West has RF=3, needs QUORUM=2
  - Works perfectly!
  - Latency: ~5-10ms in US-West

✓ Reads from US-East users → US-West:
  - US-West has complete data
  - Works perfectly!
  - Latency: ~5-10ms in US-West

✓ US-West and EU users: Unaffected

EACH_QUORUM:
❌ FAILS because can't reach US-East!
   - Needs QUORUM in EACH datacenter
   - US-East unreachable
   - Writes fail until strategy changed

QUORUM (Global):
✓ Still works:
   - Total RF = 8, QUORUM = 5
   - US-West (3) + EU-West (2) = 5
   - Can still satisfy QUORUM
   - But slower than LOCAL_QUORUM

Recovery Process:

When US-East comes back online (T=4 hours):

Phase 1: Nodes Rejoin (T=0-5 minutes)
- US-East nodes start up
- Join gossip network
- Mark themselves as available

Phase 2: Hinted Handoff (T=5-30 minutes)
- US-West replays all hints to US-East
- Catches up on missed writes
- Usually ~10K-100K writes

Phase 3: Anti-Entropy Repair (T=30 mins - 2 hours)
- nodetool repair runs automatically
- Compares data across all DCs
- Fixes any inconsistencies
- Ensures complete synchronization

Phase 4: Back to Normal (T=2-4 hours)
- US-East fully synchronized
- Traffic gradually returns to US-East
- GeoDNS routes US users back
- Latency returns to normal

Real Example: AWS US-East-1 Outage (Nov 2020):

• Affected: Netflix, Robinhood, Adobe, many others
• Netflix (Cassandra): Zero downtime
• Duration: 4 hours
• Impact: +80ms latency for US-East users, but service continued
• Recovery: Automatic when US-East returned

Best Practices for DC Failure:

• Always use LOCAL_QUORUM (EACH_QUORUM fails if any DC down)
• Minimum 3 DCs: Can lose 1 and still have 2
• Monitor Gossip: Detect failures immediately
• Test Failover: Regularly shutdown a DC to verify
• GeoDNS: Automatic traffic routing

Answer: A comprehensive multi-DC design strategy:

1. Datacenter Placement Strategy:

Geographic Coverage:
- Place DCs close to user concentrations
- Example for global app:
  * US-East (Virginia) - 40% of users
  * US-West (California) - 20% of users
  * EU-West (Ireland) - 25% of users
  * Asia-Pacific (Singapore) - 15% of users

Replication Factor per DC:
- High traffic regions: RF=3
- Medium traffic: RF=2
- Low traffic/DR only: RF=1

Example Configuration:
CREATE KEYSPACE global_app WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'US-East': 3,        # 40% users, high traffic
    'US-West': 3,        # 20% users, primary backup
    'EU-West': 3,        # 25% users, high traffic
    'Asia-Pacific': 2    # 15% users, growing
};

2. Client Routing Strategy:

GeoDNS Layer:
1. User connects to app.example.com
2. GeoDNS (Route53/Cloudflare) detects user location
3. Returns IP of nearest datacenter:
   - US users → US-East
   - EU users → EU-West
   - Asia users → Asia-Pacific

Example Route53 Config:
- Record: app.example.com
- Type: Geolocation
- US-East: 10.1.0.1 (health check enabled)
- US-West: 10.2.0.1 (failover for US-East)
- EU-West: 10.3.0.1 (health check enabled)
- Asia-Pacific: 10.4.0.1 (health check enabled)

Result: User always connects to nearest DC (5-10ms latency)

3. Consistency Level Strategy:

Default (99% of operations):
- Reads: LOCAL_QUORUM
- Writes: LOCAL_QUORUM
- Latency: 5-10ms
- Consistency: Strong within DC, eventual across DCs

Critical Operations (1%):
- Financial transactions: EACH_QUORUM
- Billing updates: EACH_QUORUM
- Latency: 150-200ms (acceptable for critical ops)

Application Code:
// Default queries
session.execute(query, ConsistencyLevel.LOCAL_QUORUM);

// Critical financial transaction
if (isCriticalTransaction) {
    session.execute(query, ConsistencyLevel.EACH_QUORUM);
}

4. Connection Pool Optimization:

DataStax Java Driver Configuration:

// Application running in US-East
Cluster cluster = Cluster.builder()
    .addContactPoints("us-east-node1", "us-east-node2")
    .withLoadBalancingPolicy(
        DCAwareRoundRobinPolicy.builder()
            .withLocalDc("US-East")              // Prefer local DC
            .withUsedHostsPerRemoteDc(2)         // Only 2 remote connections
            .build()
    )
    .withPoolingOptions(
        new PoolingOptions()
            .setConnectionsPerHost(HostDistance.LOCAL, 4, 10)   // Local: 4-10 connections
            .setConnectionsPerHost(HostDistance.REMOTE, 1, 2)   // Remote: 1-2 connections
    )
    .build();

Result:
- Queries go to local DC 99.9% of time
- Remote DCs only for failover
- Minimizes cross-DC traffic

5. Read/Write Path Optimization:

• Write-heavy workloads: LOCAL_QUORUM writes, async replication to other DCs
• Read-heavy workloads: LOCAL_QUORUM reads from nearest DC
• Mixed workloads: Partition data by geography when possible

6. Caching Strategy:

Multi-Layer Caching:
1. Application Cache (Redis/Memcached) - per DC
   - Hot data: 5-minute TTL
   - Latency: <1ms

2. Cassandra Row Cache - per DC
   - Frequently accessed rows
   - Latency: 1-2ms

3. Cassandra Query - LOCAL_QUORUM
   - Cache miss
   - Latency: 5-10ms

Result: 90%+ requests served from cache (<1ms)

7. Monitoring & Alerting:

Key Metrics to Monitor:
- Per-DC latency (p50, p95, p99)
  * Alert if p99 > 50ms for LOCAL_QUORUM
- Cross-DC replication lag
  * Alert if lag > 500ms
- Per-DC error rate
  * Alert if errors > 0.1%
- Hint pressure
  * Alert if hints queue > 10K

Tools:
- Prometheus + Grafana
- DataStax OpsCenter
- Custom dashboards per DC

Expected Latency Results:

Operation	p50	p95	p99
Cache hit	0.5ms	1ms	2ms
LOCAL_QUORUM read	5ms	12ms	25ms
LOCAL_QUORUM write	6ms	15ms	30ms

Answer: Understanding RF trade-offs per DC is crucial for cost and availability balance:

Scenario Comparison:

Scenario	Configuration	Total Copies	Storage Cost
Uniform High	US-East:3, US-West:3, EU:3	9	9x data
Mixed (Recommended)	US-East:3, US-West:3, EU:2	8	8x data (-11%)
Cost Optimized	US-East:3, US-West:2, EU:1	6	6x data (-33%)

Detailed Analysis:

1. Primary DC (RF=3) - High Traffic Region:

Benefits:
✓ Can lose 1 node and maintain QUORUM (2 of 3)
✓ Strong consistency with LOCAL_QUORUM
✓ 99.99% availability
✓ Best for user-facing DCs

Trade-offs:
✗ Higher storage cost (3x data in this DC)
✗ More nodes to manage
✗ Higher operational complexity

Use when:
- High user traffic (US-East, EU-West)
- Primary serving regions
- Critical to business

2. Secondary DC (RF=2) - Medium Traffic:

Benefits:
✓ Moderate storage cost (2x data in this DC)
✓ Can still serve with LOCAL_QUORUM (2 of 2)
✓ Good for disaster recovery
✓ Cost savings vs RF=3

Trade-offs:
✗ CANNOT lose any nodes without losing QUORUM
✗ LOCAL_QUORUM requires both nodes (less fault tolerant)
✗ If 1 node down, must use CL=ONE (weaker consistency)
✗ Lower availability than RF=3

Warning with RF=2:
If 1 of 2 nodes down:
- LOCAL_QUORUM needs 2 of 2 → FAILS!
- Must downgrade to CL=ONE
- Reduced consistency guarantee

Use when:
- Secondary regions with less traffic
- Disaster recovery backup
- Cost optimization needed
- Can tolerate reduced availability

3. DR-Only DC (RF=1) - Rarely Used:

Benefits:
✓ Lowest storage cost (1x data in this DC)
✓ Complete data copy for disaster recovery
✓ Good for compliance (data in specific region)

Trade-offs:
✗ NO redundancy within this DC
✗ Cannot use LOCAL_QUORUM (need 1, have 1)
✗ Must use CL=ONE (weakest consistency)
✗ Node failure = data unavailable in this DC
✗ Should NOT serve user traffic

Use when:
- Disaster recovery only (not serving traffic)
- Compliance requirement (must have data in region)
- Extreme cost optimization
- Other DCs have full RF=3

DO NOT use RF=1 if:
- Serving user traffic in this DC
- Need any fault tolerance
- PRIMARY serving region

Real-World Example: Netflix Configuration:

CREATE KEYSPACE viewing_history WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'US-East': 3,      # Primary - 60% of users
    'US-West': 3,      # Primary - backup and 20% users
    'EU-West': 3       # Primary - 20% of users
};
# Total: 9 copies (expensive but maximum availability)

CREATE KEYSPACE recommendation_cache WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'US-East': 3,      # Primary
    'US-West': 2,      # Secondary (can regenerate)
    'EU-West': 2       # Secondary
};
# Total: 7 copies (22% cost savings, acceptable for cache)

CREATE KEYSPACE audit_logs WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'US-East': 3,      # Primary
    'US-West': 2,      # Backup
    'EU-West': 1       # Compliance only
};
# Total: 6 copies (33% cost savings, rarely queried)

Cost Impact Example:

Scenario: 1TB of raw data

All RF=3 (Uniform):
- US-East: 3TB
- US-West: 3TB
- EU-West: 3TB
- Total: 9TB storage
- Monthly cost (AWS): ~$900 (9TB × $100/TB)

Mixed RF (Recommended):
- US-East: 3TB (RF=3)
- US-West: 3TB (RF=3)
- EU-West: 2TB (RF=2)
- Total: 8TB storage
- Monthly cost: ~$800 (8TB × $100/TB)
- Savings: $100/month, $1,200/year per TB

Cost Optimized:
- US-East: 3TB (RF=3)
- US-West: 2TB (RF=2)
- EU-West: 1TB (RF=1)
- Total: 6TB storage
- Monthly cost: ~$600 (6TB × $100/TB)
- Savings: $300/month, $3,600/year per TB

But: Reduced availability in US-West and EU-West!

Decision Matrix:

• Use RF=3: Primary user-facing DCs, critical data, high traffic
• Use RF=2: Secondary DCs, backup regions, cost-conscious deployments
• Use RF=1: DR only, compliance requirements, NEVER for serving traffic

Key Principle: Always have at least 2 DCs with RF≥3 to survive complete DC failure!