Cassandra Peer-to-Peer Model

Peer-to-Peer Architecture: A distributed system design where all nodes have equal responsibilities and capabilities. No node has special authority over others.

Key Differences from Master-Slave:

Master-Slave:

• Master handles all writes, slaves handle reads
• Clear hierarchy with master having special role
• If master fails, system needs failover/promotion
• Write scalability limited by master capacity
• Example: MySQL with read replicas

Peer-to-Peer (Cassandra):

• All nodes handle both reads and writes
• Flat organization, no hierarchy
• Node failure transparent, no failover needed
• Write scalability by adding more peers
• Example: Cassandra, Riak, DynamoDB

Interview Tip: Emphasize that P2P trades strong consistency for availability and scalability - perfect for CAP theorem discussion!

Cassandra uses several mechanisms:

1. Temporary Coordinator Role:

• Any node can act as coordinator for a client request
• Coordinator role is per-request, not permanent
• Coordinator determines which nodes own data (via hash function)
• Orchestrates reads/writes to appropriate replicas

2. Gossip Protocol:

• Nodes exchange information peer-to-peer
• Every node knows cluster topology (ring, token ranges)
• Information spreads exponentially (reaches all nodes quickly)
• No central authority needed for cluster state

3. Consistent Hashing:

• Deterministic algorithm decides data placement
• All nodes use same algorithm, get same result
• No need to ask "master" where data lives

4. Quorum-Based Consensus:

• Operations require majority agreement (e.g., 2 out of 3 nodes)
• No single node makes unilateral decisions
• Prevents conflicts during network partitions

Example Flow:

Gossip Protocol: A peer-to-peer communication mechanism where nodes periodically exchange information with random neighbors, causing information to spread exponentially throughout the cluster.

How It Works:

• Every 1 second, each node picks 1-3 random nodes to "gossip" with
• They exchange: cluster membership, ring topology, schema, load info
• Each node updates its local view based on what it learns
• Information spreads exponentially: 1 → 2 → 4 → 8 → 16 nodes per round

Why Essential for P2P:

1. Distributed Failure Detection:

• No master to monitor node health
• Nodes detect failures by missing gossip messages
• Failure information spreads to all nodes quickly

2. Cluster State Synchronization:

• All nodes maintain consistent view of ring topology
• When node joins/leaves, gossip spreads the update
• Eventually consistent cluster state

3. Schema Propagation:

• CREATE TABLE on one node spreads to all nodes via gossip
• No need to manually update each node

Mathematical Efficiency:

Analogy: Like spreading rumors in an office - each person tells 2 friends, who each tell 2 friends. Everyone knows within minutes, no central announcement needed!

Coordinator Node: The node that receives a client request and orchestrates its execution across the cluster. Important: This is a temporary role for that specific request only.

Key Characteristics:

• Not Permanent: ANY node can be coordinator depending on which one client connects to
• Per-Request: Node A coordinates Request 1, Node B coordinates Request 2
• No Special Hardware: Doesn't require more resources than other nodes
• Load Balanced: Coordinator role distributed across all nodes over time

Coordinator Responsibilities:

For Writes:

• Hash partition key to determine replica nodes
• Send write to all replica nodes in parallel
• Wait for acknowledgments based on consistency level
• Return success/failure to client
• If replica unavailable, store hint for later

For Reads:

• Determine which nodes have the data
• Send read request to appropriate replicas
• Wait for responses based on consistency level
• Compare timestamps if multiple responses
• Perform read repair if data inconsistent
• Return data to client

Example Scenario:

Why This Matters: Unlike master-slave where master is bottleneck, coordinator role is distributed. If coordinator fails mid-request, client simply retries with different node.

P2P's Natural Multi-DC Support:

Peer-to-peer architecture extends naturally to multiple geographic datacenters because there's no "master" that must be in one location.

How It Works:

1. Logical Rings Per Datacenter:

• Nodes in each DC form their own ring
• Example: US-East ring, US-West ring, EU-West ring
• All rings are part of same cluster

2. NetworkTopologyStrategy:

3. Local Coordinator Preference:

• Client in London connects to EU-West nodes
• EU node becomes coordinator (local)
• Writes go to local replicas first (fast)
• Async replication to other DCs (eventually consistent)

4. LOCAL_QUORUM Consistency:

• Only requires quorum within local DC
• Example: 2 out of 3 nodes in EU-West
• No waiting for cross-DC latency
• Sub-10ms operations even globally distributed

Benefits vs Master-Slave:

• No Master DC: Master-slave requires master in one DC (what if that DC fails?)
• Local Writes: P2P allows writes in any DC, master-slave requires remote master
• DC Failure: P2P continues in other DCs, master-slave requires failover
• Latency: P2P enables local operations, master-slave has cross-DC latency

Real Example - Global E-Commerce:

Network Partition: When network fails and cluster splits into isolated groups that cannot communicate.

The Split-Brain Problem:

Cassandra's Solution: Quorum-Based Operations

How It Works:

• QUORUM requires majority of replicas (N/2 + 1)
• Only ONE partition can contain a majority
• Minority partition cannot achieve quorum → Rejects writes

Example with RF=3, QUORUM writes:

Important Notes:

• Reads with consistency ONE still work in minority partition (stale data acceptable)
• This trades availability (minority can't write) for consistency (no conflicting writes)
• CAP theorem in action: During partition, choose CP (Consistency + Partition-tolerance) over AP

Interview Insight: This demonstrates Cassandra is NOT "always available" - it's "tunable availability" based on consistency level chosen!

The Surprising Answer: P2P is operationally simpler despite seeming more complex conceptually.

Master-Slave Operational Challenges:

• Failover Choreography:
  • Detect master failure
  • Elect new master (consensus algorithm)
  • Promote slave to master
  • Update application connection strings
  • Risk of split-brain if network partition
  • Typical downtime: 30 seconds to 5 minutes
• Adding Capacity:
  • Can only add read replicas (slaves)
  • Write capacity still limited by master
  • Must upgrade master hardware for more writes (vertical scaling = expensive)
• Upgrades:
  • Must upgrade master first (downtime)
  • Or promote slave, upgrade old master, demote back (complex)

P2P Operational Simplicity:

• Node Failure:
  • Do nothing - system continues automatically
  • Replicas handle requests
  • Hinted handoff ensures no data loss
  • Replace failed node when convenient
  • Zero downtime
• Adding Capacity:
  • Add node: `nodetool join`
  • Data automatically rebalances
  • Increases both read AND write capacity
  • Zero downtime, happens in background
• Upgrades:
  • Rolling restart: upgrade one node at a time
  • Each node down only seconds
  • Cluster continues serving traffic
  • Can do during business hours

Real-World Operations Comparison:

DevOps Feedback: "With master-slave, I'm on-call for failovers. With Cassandra P2P, I sleep well - cluster self-heals."

Short Answer: Generally NO for core banking transactions, but YES for specific banking use cases.

Where P2P (Cassandra) is NOT Appropriate:

Core Banking Transactions:

• Requirement: Strong ACID guarantees, immediate consistency
• Example: Account balance updates, fund transfers
• Why P2P fails:
  • Eventual consistency means balance might be temporarily inconsistent
  • No multi-row transactions across partitions
  • Risk of double-spending if not careful
• Better choice: PostgreSQL, Oracle RAC with strong consistency

Where P2P (Cassandra) IS Appropriate in Banking:

1. Fraud Detection System:

• Use case: Log all transactions for ML analysis
• Why P2P works:
  • Write-heavy (millions of transactions/sec)
  • Eventual consistency OK (fraud detection is async)
  • Need to scale globally
  • Must never lose transaction logs (RF=3 ensures durability)

2. Transaction History/Audit Logs:

• Use case: Store completed transactions for customer view
• Why P2P works:
  • Read-heavy, immutable data
  • Partition by customer_id (perfect for Cassandra)
  • 99.99% availability critical (customers checking balances 24/7)
  • Scale to billions of transactions

3. User Session Management:

• Use case: Track logged-in users across mobile app
• Why P2P works:
  • High availability critical (login always works)
  • Temporary data (sessions expire)
  • Geographic distribution (global banking)

Hybrid Architecture Recommendation:

Interview Answer Structure:

1. Acknowledge trade-offs (P2P = availability > consistency)
2. Explain why core banking needs strong ACID (PostgreSQL better)
3. Identify specific banking use cases where P2P excels
4. Recommend hybrid architecture using both
5. Show understanding of CAP theorem and system design trade-offs

Bonus Points: Mention that some banks DO use Cassandra extensively (Capital One, ING) but for specific workloads, not core transaction processing.