Cassandra Network Topology

Answer:

Network Topology is the hierarchical organization of Cassandra nodes into datacenters and racks, enabling geographic distribution and fault isolation.

Three-Level Hierarchy:

• Cluster: Top level - entire deployment
• Datacenter: Geographic or logical grouping (e.g., US-EAST, EU-WEST)
• Rack: Failure domain within a datacenter (e.g., Rack-A, Rack-B)

Why It's Critical:

• Low Latency: Serve users from nearest datacenter (Tokyo users → ASIA-PAC DC = 5ms vs US-EAST = 250ms)
• Fault Isolation: Power failure on Rack-A doesn't affect Rack-B and Rack-C
• Disaster Recovery: Entire datacenter can fail, others continue serving
• Data Sovereignty: EU data stays in EU (GDPR compliance)
• Smart Replication: NetworkTopologyStrategy spreads replicas across racks/DCs

Answer:

A Snitch is a component that determines the datacenter and rack location of each node, enabling NetworkTopologyStrategy to make intelligent replica placement decisions.

What Snitch Provides:

• Datacenter Name: Which DC does this node belong to? (e.g., US-EAST)
• Rack Name: Which rack within the DC? (e.g., RACK-A)
• Relative Distance: How "close" are two nodes for routing decisions
• Query Routing: Which node should handle a request (prefer local)

Without Snitch:

• All nodes treated as same location
• Replicas might all go to same rack (no fault tolerance!)
• Queries routed randomly (high latency)
• Can't enforce data sovereignty

With Snitch:

• Replicas intelligently spread across racks and DCs
• Queries routed to nearest node (low latency)
• Data stays in correct geographic region
• Survives rack and DC failures gracefully

Answer:

GossipingPropertyFileSnitch:

• Configuration: Read from cassandra-rackdc.properties file on each node
• Setup: Manual - you specify dc=US-EAST, rack=RACK-A
• Discovery: Gossip protocol shares topology with cluster
• Flexibility: Works anywhere - on-prem, any cloud, hybrid
• Use When: 95% of deployments, any environment, need full control
• Pros: Simple, flexible, self-healing, cloud-agnostic
• Cons: Manual file editing required

Ec2Snitch / Ec2MultiRegionSnitch:

• Configuration: Auto-detects from AWS EC2 metadata API
• Setup: Zero configuration required
• Mapping: AWS Region → Datacenter, Availability Zone → Rack
• Example: us-east-1a becomes DC: us-east-1, Rack: 1a
• Use When: 100% AWS deployment, want zero-config
• Pros: No manual config, AWS-native, auto-updates
• Cons: AWS only, uses public IPs (Ec2Snitch), not portable

Recommendation:

Use GossipingPropertyFileSnitch even on AWS for better control and portability. Only use Ec2Snitch if you absolutely need zero-config and will never migrate off AWS.

Answer:

SimpleStrategy:

• Topology Awareness: None - treats all nodes as equal
• Replication: Places replicas clockwise on ring, ignoring DC/rack
• Configuration: Single RF for entire cluster
• Example: {'class': 'SimpleStrategy', 'replication_factor': 3}
• Multi-DC Support: Not supported - can't specify per-DC RF
• Fault Tolerance: Low - all replicas might be on same rack
• Production Use: ❌ NEVER - only for testing/dev

NetworkTopologyStrategy:

• Topology Awareness: Fully aware of datacenters and racks
• Replication: Intelligently spreads replicas across different racks
• Configuration: Per-datacenter RF specification
• Example: {'class': 'NetworkTopologyStrategy', 'US-EAST': 3, 'EU-WEST': 2}
• Multi-DC Support: Fully supported - different RF per DC
• Fault Tolerance: High - guarantees replicas on different racks/DCs
• Production Use: ✅ ALWAYS - even with single DC

Key Difference:

NetworkTopologyStrategy ensures replicas are on different racks within each DC. With RF=3 and 3 racks, SimpleStrategy might put all 3 replicas on Rack-A (disaster!), while NetworkTopologyStrategy guarantees one replica per rack (safe!).

Answer:

NetworkTopologyStrategy uses an intelligent algorithm to ensure replicas are on different racks:

Algorithm Steps:

1. Start: Primary replica placed based on partition key hash (token)
2. Walk Ring: Move clockwise around token ring
3. Check Rack: Is next node on same rack as existing replica?
4. Skip or Place: If same rack → skip, if different rack → place replica
5. Repeat: Continue until RF replicas placed in this DC
6. Next DC: Repeat entire process for next datacenter

Example (RF=3, 3 racks):

• Primary: Token hash → Node 5 (Rack-B)
• Walk ring: Node 6 (Rack-B) ❌ Skip! Same rack as Node 5
• Continue: Node 7 (Rack-C) ✅ Place replica 2 (different rack)
• Continue: Node 8 (Rack-C) ❌ Skip! Same rack as Node 7
• Continue: Node 1 (Rack-A) ✅ Place replica 3 (different rack)
• Result: Replicas on Rack-A, Rack-B, Rack-C ✅

Guarantee:

With 3+ racks and RF=3, you're guaranteed one replica per rack. If rack fails, other 2 racks still have complete data!

Answer:

WARNING: This is EXTREMELY dangerous and can cause data loss!

What Goes Wrong:

• Topology Confusion: Cassandra thinks nodes moved to different DCs/racks
• Replication Breaks: Wrong replicas might be deleted, new ones created
• Data Loss Risk: Old replicas dropped before new ones fully replicated
• Query Failures: Can't find correct replicas, routing errors
• Consistency Issues: Cluster becomes inconsistent, repair can't fix
• Split Brain: Different nodes think data is in different places

Safe Migration Procedure:

1. Plan: Document current topology, plan new topology
2. Backup: Take full snapshots of entire cluster
3. Repair: Run full cluster repair first (nodetool repair)
4. Maintenance Window: Schedule downtime if possible
5. One Node: Change snitch on ONE node only
6. Update Config: Modify cassandra.yaml and cassandra-rackdc.properties
7. Restart: Restart that one node
8. Verify: Check topology: nodetool status, nodetool describecluster
9. Monitor: Watch logs for errors, check metrics
10. Iterate: If correct, proceed to next node. If wrong, REVERT!
11. Final Repair: After all nodes migrated, run full repair

Best Practice:

Get snitch configuration right from the start! Changing it later is risky and time-consuming.

Answer:

Rack awareness is Cassandra's ability to understand physical failure domains (racks) and distribute replicas accordingly to prevent correlated failures.

Physical Rack Concept:

• Physical Rack: Literal server rack with shared power supply and network switch
• Failure Domain: If power fails or switch breaks, entire rack goes offline
• Cloud Equivalent: Availability Zone in AWS/GCP/Azure

Without Rack Awareness:

• All 3 replicas (RF=3) might be on same rack
• Power failure on that rack = lose ALL 3 copies = DATA LOST!
• Network switch failure = all replicas unreachable = OUTAGE!
• Single point of failure despite having replicas

With Rack Awareness (NetworkTopologyStrategy):

• Replicas automatically spread across different racks
• RF=3 + 3 racks = one replica per rack guaranteed
• Power failure on Rack-A = Rack-B and Rack-C still serve data
• Can lose ANY single rack without data loss or downtime

Real-World Example:

Amazon 2017 Power Outage: PDU failure in us-east-1a

• No Rack Awareness: All 3 nodes in us-east-1a → 4-hour outage, $500K/hour loss
• With Rack Awareness: Nodes in us-east-1a, 1b, 1c → Zero downtime! Other AZs served traffic

Best Practice:

Minimum 3 racks per datacenter. In cloud: map each rack to different Availability Zone.

Answer:

Use NetworkTopologyStrategy with per-datacenter replication factor specification:

Configuration Example:

Why Different RFs?

• US-EAST (RF=3): Primary datacenter, most users, highest traffic
• EU-WEST (RF=2): Secondary datacenter, moderate traffic
• ASIA-PAC (RF=2): Read-only analytics DC, lower requirements

Data Distribution:

• User record for user_id='alice' stored in 7 nodes total:
• US-EAST: Node 5 (Rack-A), Node 12 (Rack-B), Node 19 (Rack-C)
• EU-WEST: Node 34 (Rack-A), Node 41 (Rack-B)
• ASIA-PAC: Node 56 (Rack-A), Node 63 (Rack-B)

Consistency Levels:

• Writes: LOCAL_QUORUM (2 nodes in local DC)
• Reads: LOCAL_ONE (1 node in local DC, ultra-fast)
• Critical: EACH_QUORUM (QUORUM in every DC)

Verification:

Answer:

prefer_local controls whether Cassandra prefers reading from replicas in the local datacenter.

Configuration:

When prefer_local=true (RECOMMENDED):

• Queries prefer replicas in same datacenter
• Lower latency (1-5ms vs 100-200ms cross-DC)
• Reduced WAN bandwidth costs
• Better user experience (faster responses)
• Only contacts remote DC if local replicas unavailable

When prefer_local=false:

• Queries may go to any datacenter
• Higher latency (unpredictable)
• Higher WAN costs
• Useful for testing or special analytics workloads

Example Impact:

• User in New York, US-EAST DC:
• prefer_local=true: Read from US-EAST replica (2ms latency)
• prefer_local=false: Might read from EU-WEST replica (150ms latency!)

Interaction with Consistency Levels:

• LOCAL_QUORUM: Always reads from local DC (prefer_local irrelevant)
• QUORUM: Uses prefer_local to choose which DC's replicas
• ONE: Uses prefer_local to pick replica location

Best Practice:

Always set prefer_local=true in production. Only set false for specific analytics/reporting workloads.

Answer:

Cassandra automatically routes queries to healthy datacenters when one DC fails, with behavior depending on consistency level:

Scenario: US-EAST datacenter goes offline

With LOCAL_QUORUM (RECOMMENDED):

• US-EAST clients: Queries FAIL (local DC down, can't reach LOCAL_QUORUM)
• Solution: Load balancer redirects US-EAST traffic to US-WEST
• US-WEST clients: Unaffected, continue serving from US-WEST replicas
• EU-WEST clients: Unaffected, continue serving from EU-WEST replicas
• Benefit: Controlled failover, no cross-DC latency impact

With QUORUM (Global):

• All queries continue working!
• Needs (Total RF / 2) + 1 replicas globally
• Example: RF=9 (3 per DC × 3 DCs) → QUORUM=5
• US-EAST down (3 replicas lost) → 6 replicas remain → Still achieve QUORUM
• Downside: Higher latency (might query remote DCs)

With EACH_QUORUM:

• All writes FAIL!
• Requires QUORUM in EVERY datacenter
• US-EAST down → Can't reach QUORUM there → Write rejected
• Use case: Strong consistency, but poor availability

Automatic Failover Example:

Recovery Process:

1. Detection: Gossip protocol detects US-EAST nodes down (~30 seconds)
2. Routing: Coordinator stops sending queries to US-EAST
3. Hinted Handoff: Writes queued for US-EAST nodes
4. DC Restored: Hints replayed, nodes catch up
5. Repair: Run nodetool repair to ensure consistency

Best Practice:

Use LOCAL_QUORUM for writes and LOCAL_ONE for reads. Set up load balancer to redirect traffic from failed DC to nearest healthy DC.

Answer:

Always use NetworkTopologyStrategy, even with single DC:

• Rack Awareness: Still need fault isolation within DC (different racks/AZs)
• Future Expansion: Easy to add DCs later without changing keyspace
• Best Practice: Consistent approach across all environments
• No Downside: Same performance as SimpleStrategy in single DC
• Production Ready: SimpleStrategy is for testing only

Even with single DC, use: {'class': 'NetworkTopologyStrategy', 'DC1': 3}

Answer:

Cloud Availability Zones (AZs) are perfect Cassandra racks:

• AWS: us-east-1a → DC: us-east-1, Rack: 1a
• GCP: us-central1-a → DC: us-central1, Rack: a
• Azure: eastus-1 → DC: eastus, Rack: 1
• Isolation: AZs have separate power, cooling, networking
• Best Practice: Deploy nodes across 3+ AZs in each region
• Configuration: Set rack=1a, rack=1b, rack=1c in cassandra-rackdc.properties

With 3 AZs and RF=3, you get one replica per AZ - survives any AZ failure!

Answer:

Minimum: 3 racks per datacenter

• With RF=3: One replica per rack guaranteed
• Fault Tolerance: Can lose any 1 rack, still have QUORUM (2/3)
• With 2 Racks: Can't survive single rack failure with QUORUM
• With 1 Rack: No fault isolation at all!

Math:

• 3 racks, RF=3: Rack fails → 2 replicas remain → QUORUM(2) achieved ✅
• 2 racks, RF=3: Rack fails → 1-2 replicas remain → QUORUM(2) might fail ❌

Answer:

Data Sovereignty: Legal requirement that data must stay within specific geographic boundaries.

• GDPR (EU): EU citizen data must stay in EU
• Russia: Russian citizen data must stay in Russia
• China: Chinese data must stay in China

How Cassandra Helps:

• Datacenter Isolation: Create separate EU-WEST datacenter
• Keyspace Design: EU users → eu_users keyspace with RF only in EU-WEST
• No Cross-DC: EU data never replicates to US or Asia DCs
• Local Queries: EU queries only hit EU-WEST nodes

Configuration Example:

Answer:

Scenario: Global e-commerce with 100M users, 1M products, 10K orders/second

Datacenter Design:

• US-EAST: 100 nodes (3 racks × 33 nodes) - Primary region
• US-WEST: 100 nodes (3 racks × 33 nodes) - DR + West coast
• EU-WEST: 60 nodes (3 racks × 20 nodes) - European customers
• ASIA-PAC: 40 nodes (3 racks × 13 nodes) - Asian customers
• Total: 300 nodes across 4 continents

Keyspace Strategy:

Consistency Levels:

• Products: Write EACH_QUORUM, Read LOCAL_ONE (eventual consistency OK)
• Sessions: Write LOCAL_QUORUM, Read LOCAL_ONE (fast, local)
• Orders: Write QUORUM, Read QUORUM (strong consistency)

Answer:

Two Files Required:

1. cassandra.yaml:

2. cassandra-rackdc.properties:

Important Notes:

• Both files in /etc/cassandra/ directory (Debian/Ubuntu) or /etc/dse/cassandra (DSE)
• Requires node restart after changes
• DC/rack names case-sensitive and permanent
• Each node can have different dc/rack values

Answer:

Gossip protocol is how nodes discover and share topology information:

• Every Second: Each node picks 1-3 random nodes to gossip with
• Exchange Info: Share state: DC/rack, health, schema, load
• Propagation: Information spreads exponentially through cluster
• Convergence: Entire cluster knows topology within ~30 seconds
• Self-Healing: Automatically detects and shares node changes

Example:

• Node 1 starts with dc=US-EAST, rack=RACK-A
• Node 1 gossips to Node 5: "I'm in US-EAST/RACK-A"
• Node 5 gossips to Node 12: "Node 1 is US-EAST/RACK-A"
• After few rounds: All nodes know Node 1's location

Answer:

Key Commands:

1. nodetool status:

2. nodetool describecluster:

3. nodetool ring:

4. CQL DESCRIBE:

Answer:

YES! Different DCs can have different node counts - this is common:

Common Scenarios:

• Primary/Secondary DCs:
  • US-EAST (primary): 100 nodes
  • US-WEST (DR): 50 nodes
  • Different workloads justify different sizes
• Analytics DC:
  • ONLINE (user-facing): 200 nodes
  • ANALYTICS (batch jobs): 50 nodes
  • Analytics runs overnight, needs fewer nodes
• Regional Traffic:
  • US-EAST: 150 nodes (high traffic)
  • EU-WEST: 100 nodes (moderate traffic)
  • ASIA-PAC: 50 nodes (growing market)

Important Considerations:

• Token Distribution: Vnodes auto-adjust for different sizes
• Minimum Nodes: Must have ≥ RF nodes in each DC for that keyspace
• Load Balance: Smaller DC = more data per node

Example:

Answer:

DISASTER! This breaks replication and can cause data loss.

Example Misconfiguration:

What Goes Wrong:

• Multiple DCs Created: Cassandra sees 3 DCs: "US-EAST", "us-east", "USEAST"
• Replication Breaks: Keyspace expects RF=3 in "US-EAST", but only 2 nodes match!
• Writes Fail: Can't achieve QUORUM (need 2/3, only have 2 nodes in "US-EAST")
• Data Loss Risk: Some nodes not receiving replicas
• Query Routing Broken: Coordinator can't find enough replicas

Prevention:

• Automation: Use config management (Ansible, Chef, Puppet)
• Validation: Verify with nodetool status before adding to production
• Standards: Document exact DC/rack naming convention
• Case Sensitive: Use consistent capitalization (recommend ALL-CAPS)

Recovery:

• Fix cassandra-rackdc.properties on misconfigured nodes
• Restart nodes one at a time
• Run nodetool repair on entire cluster
• Verify topology with nodetool describecluster

Answer:

Rack-aware routing optimizes query performance by preferring replicas on nodes within the same rack:

• Lower Latency: Same rack = same network switch = sub-millisecond latency
• Less Network Hops: Avoid top-of-rack switch → core switch → another rack
• Higher Bandwidth: Within-rack links typically faster than cross-rack
• Cost Savings: Reduced cross-rack network traffic

Example: Coordinator on RACK-A prefers reading from RACK-A replica (0.5ms) vs RACK-B replica (2ms).

Answer:

YES! Common pattern for workload separation:

• Physical DCs: US-EAST, EU-WEST (geographic locations)
• Logical DCs: ANALYTICS, SEARCH (same physical location as US-EAST)
• Use Case: Separate heavy analytics from online traffic

Example Configuration:

Benefit: Heavy analytics queries don't impact online user experience!

Answer:

Topology determines which nodes store hints when replicas are unavailable:

• Same DC Hints: Hints stored on nodes in same datacenter (by default)
• Rack Awareness: Hints prefer different rack than down node
• Cross-DC Hints: Disabled by default (configure with hinted_handoff_enabled)

Example:

• Write for Node 5 (US-EAST/RACK-A) but it's down
• Hint stored on Node 12 (US-EAST/RACK-B) - same DC, different rack
• When Node 5 returns, Node 12 replays hints

Answer:

Safe Migration Steps:

1. Check Current: DESCRIBE KEYSPACE my_keyspace;
2. Run Repair: nodetool repair -full my_keyspace
3. Alter Keyspace:
   ALTER KEYSPACE my_keyspace WITH replication = {
     'class': 'NetworkTopologyStrategy',
     'DC1': 3 -- Same RF as before
   };
4. Repair Again: nodetool repair -full my_keyspace
5. Verify: nodetool status my_keyspace

Important: No downtime required! Replication happens asynchronously.

Answer:

Comprehensive DR Strategy:

1. Multi-DC Architecture:

• Primary: US-EAST (200 nodes, 3 racks)
• DR: US-WEST (200 nodes, 3 racks) - mirror of primary
• Remote DR: EU-WEST (100 nodes, 3 racks) - geographic diversity

2. Replication Configuration:

3. Consistency Levels:

• Normal: Write LOCAL_QUORUM, Read LOCAL_ONE
• DR Event: Switch to QUORUM (cross-DC) or redirect traffic

4. Failover Plan:

• Detection: Monitor US-EAST health (30 seconds to detect)
• DNS Update: Point users to US-WEST (TTL=60s)
• Traffic Shift: Load balancer redirects within 2 minutes
• RTO: 5 minutes (Recovery Time Objective)
• RPO: 0 seconds (no data loss - synchronous replication)

5. Testing:

• Quarterly DR drills - simulate US-EAST failure
• Verify US-WEST can handle full load
• Test failback procedures