The CAP Theorem Explained

Answer:

The CAP Theorem (Brewer's Theorem) states that a distributed database system can only guarantee TWO out of THREE properties:

• Consistency (C): All nodes see the same data at the same time. Every read gets the most recent write or an error.
• Availability (A): Every request receives a response (success or failure). System is always responsive.
• Partition Tolerance (P): System continues operating despite network failures between nodes.

Key Insight: Since network partitions WILL happen in distributed systems, Partition Tolerance is mandatory. This means you must choose between Consistency or Availability during a partition.

Created by: Eric Brewer in 2000, formally proven by Seth Gilbert and Nancy Lynch in 2002.

Answer:

During a network partition, you face an impossible choice:

Scenario:

• Node A and Node B can't communicate (network partition)
• Client writes value=100 to Node A
• Node B still has old value=50
• Another client reads from Node B

The Impossible Choice:

• Option 1 (Choose C): Refuse to return anything until nodes sync → System unavailable (lose A)
• Option 2 (Choose A): Return the old value=50 → System inconsistent (lose C)

Mathematical Proof: You cannot simultaneously:

• Always respond (A)
• Always return latest data (C)
• While nodes can't communicate (P)

This is a logical impossibility, not a technology limitation!

Answer:

CP System chooses Consistency + Partition Tolerance over Availability.

Behavior During Partition:

• Refuses requests when it can't guarantee latest data
• Returns errors rather than stale data
• Waits for nodes to sync before responding
• Prioritizes correctness over uptime

Examples:

• HBase: Strong consistency, single master
• MongoDB: Primary-secondary, strong consistency default
• Redis (Sentinel mode): Coordinated failover
• Zookeeper: Consensus-based coordination

Use Cases:

• Banking and financial systems
• Inventory management
• Booking systems (flights, hotels)
• Stock trading platforms

Trade-off: System may be unavailable during network issues, but data is always accurate.

Answer:

AP System chooses Availability + Partition Tolerance over Consistency.

Behavior During Partition:

• Always responds to requests
• May return stale data temporarily
• Accepts writes on both sides of partition
• Eventually reconciles when partition heals

Examples:

• Cassandra: Tunable consistency, AP by default
• DynamoDB: Eventual consistency default
• Riak: Always available
• CouchDB: Master-master replication

Use Cases:

• Social media feeds and likes
• Shopping carts
• Session storage
• Analytics and metrics
• DNS systems

Trade-off: Data might be temporarily inconsistent, but system never goes down.

Answer:

Partition Tolerance is mandatory because network failures WILL happen in any distributed system:

Real Causes of Partitions:

• Hardware Failures: Network cables cut, switches die, routers fail
• Software Bugs: Firewall misconfiguration, router firmware bugs
• Overload: Network congestion drops packets
• Geographic: Undersea cables damaged, satellite links interrupted
• Cloud Issues: AWS availability zone failures

Why You Can't Avoid It:

• You don't control the entire network path
• Multiple points of failure between datacenters
• Even Google and Amazon experience partitions
• Can last seconds to hours

If You Don't Handle Partitions:

• Entire system goes down
• Data corruption possible
• Split-brain scenarios

Therefore: P is not optional. Real choice is between C or A!

Answer:

CA systems can only exist in theory, not in practice for distributed systems.

Single-Node Databases (CA):

• Traditional RDBMS on one server (PostgreSQL, MySQL)
• Provides both Consistency and Availability
• No partitions because there's no network!
• Problem: Not a distributed system, single point of failure

Why CA Doesn't Work in Distributed Systems:

• The moment you distribute data across multiple nodes, network partitions become possible
• You cannot prevent network failures
• Must choose how to handle partitions (C or A)

Attempts at CA:

• Two-phase commit: Tries to be CA but blocks during partitions (effectively becomes CP)
• Synchronous replication: Also blocks during partitions (CP behavior)

Conclusion: In distributed systems, CA is impossible. CAP Theorem really means "choose CP or AP"!

Answer:

Cassandra is AP by default but offers tunable consistency, allowing you to choose CP or AP per query!

AP Behavior (Default):

• Consistency Level: ONE - Reads/writes to single replica
• Always available, even during partitions
• May return stale data temporarily
• Eventually consistent

CP Behavior (When Configured):

• Consistency Level: QUORUM or ALL
• Waits for majority/all replicas
• Refuses requests if quorum unavailable
• Guarantees consistency

Example Tuning:

// AP: Fast, eventually consistent
SELECT * FROM users WHERE id = 123
  USING CONSISTENCY ONE;

// CP: Strong consistency, may fail during partition
SELECT * FROM orders WHERE id = 456
  USING CONSISTENCY QUORUM;

Best of Both Worlds: Use AP for non-critical data (views, likes) and CP for critical data (orders, payments) in the same application!

Answer:

Eventual Consistency is a consistency model used by AP systems where, if no new updates occur, all replicas will eventually converge to the same value.

How It Works:

• Writes accepted immediately on available nodes
• Asynchronous replication to other nodes
• Different nodes may have different values temporarily
• Eventually (typically milliseconds to seconds), all nodes consistent

Example:

• User updates status on Facebook
• Write goes to US datacenter at 10:00:00
• Friend in Europe reads at 10:00:01 → sees old status
• Friend reads again at 10:00:05 → sees new status
• System "eventually" became consistent!

Benefits:

• High availability (always responsive)
• Low latency (no waiting for sync)
• Partition tolerance

When It's Acceptable:

• Social media (likes, views, follows)
• Shopping carts
• Product catalogs
• Metrics and analytics

When It's NOT Acceptable:

• Financial transactions
• Inventory counts
• Booking systems

Answer:

Banking system must choose CP (Consistency + Partition Tolerance) over Availability.

Architecture Decisions:

• Database: Use CP system (PostgreSQL with sync replication, or MongoDB)
• Transactions: ACID guarantees mandatory
• Replication: Synchronous replication (wait for all replicas)
• Consistency Level: QUORUM or ALL

Handling Partitions:

• Reject transactions when partition detected
• Show "Service Temporarily Unavailable"
• Never show incorrect balance
• Never allow double-spending

Hybrid Approach (Best Practice):

• Critical Data (CP): Account balances, transactions → PostgreSQL
• Non-Critical (AP): User preferences, notifications → Cassandra
• Caching (AP): Recent transactions → Redis

Why CP?

• Wrong balance = lost customer trust
• Double-spending = financial loss
• Regulatory compliance requires accuracy
• Brief downtime acceptable, incorrect data is not!

Answer:

Strong Consistency (CP Systems):

• Every read returns the most recent write
• All replicas synchronized before response
• No stale data ever visible
• May block during partitions
• Example: MongoDB with read concern "majority"

Eventual Consistency (AP Systems):

• Reads may return stale data temporarily
• Replicas synchronized asynchronously
• Eventually converge to same value
• Never blocks
• Example: Cassandra with consistency level ONE

Comparison Example:

• Time 10:00:00: Write value=100 to Node A
• Time 10:00:01: Read from Node B
  • Strong: Returns 100 OR error (never old value)
  • Eventual: Might return old value, updates soon

Performance Impact:

• Strong: Slower (sync overhead), less available
• Eventual: Faster (no waiting), always available

Answer:

Amazon chose AP (Availability + Partition Tolerance) for shopping carts, prioritizing customer experience over perfect consistency.

Design Decision:

• Shopping cart must ALWAYS work, even during network failures
• Better to have duplicate items than lost sales
• Created DynamoDB specifically for this!

How It Works:

• User adds item → accepted immediately (no waiting)
• Write replicated asynchronously
• During partition, both sides accept writes
• When partition heals, merge carts (might have duplicates)

Example Scenario:

• Network splits US and EU datacenters
• User adds iPhone in US: Cart = [iPhone]
• Same user adds AirPods in EU: Cart = [AirPods]
• Partition heals: Cart = [iPhone, AirPods] (merged)
• Potential duplicate resolved at checkout

Why This Works:

• Users expect cart to work always
• Duplicate items easily removed
• Lost sale worse than cart confusion
• Checkout has strong consistency for payment!

Lesson: Different parts of app need different consistency! Cart is AP, payment is CP!

Answer:

While CAP Theorem is fundamental, it has several limitations and nuances:

1. Binary Choice Oversimplification:

• CAP presents C and A as binary (all or nothing)
• Reality: Consistency and availability exist on spectrums
• Modern databases offer tunable trade-offs

2. Doesn't Address Latency:

• CAP ignores performance and latency
• PACELC Theorem extends CAP to include latency
• Real-world: Latency often more important than pure availability

3. Only Applies During Partitions:

• CAP trade-off only matters when network partition occurs
• Most of the time, networks work fine
• Modern systems focus on normal operation, not just failures

4. Oversimplifies Consistency Models:

• Many consistency levels: strong, weak, causal, eventual
• CAP only discusses strong consistency
• Real systems use sophisticated consistency models

5. Doesn't Cover All Requirements:

• Doesn't address: durability, security, scalability limits
• Real systems have many more constraints

Modern Understanding:

• Use CAP as starting point, not absolute rule
• Consider PACELC: if Partition (choose A or C), else (choose Latency or Consistency)
• Design for common case (no partition) while handling failures