READ REPAIR - Automatic Data Healing & Consistency

Complete Answer:

Read Repair is Cassandra's automatic consistency mechanism that detects and fixes data inconsistencies during read operations. It's a self-healing process that ensures eventual consistency across replicas.

Why It's Needed - Inconsistencies Are Inevitable:

1. Network Partitions: Writes don't reach all replicas due to network issues
2. Node Failures: Replica is down during write operation
3. Hinted Handoff Failures: Hints expire before delivery (3 hour timeout)
4. Partial Writes: Write succeeds on 2/3 replicas, fails on 1
5. Clock Skew: Different timestamps causing Last-Write-Wins conflicts
6. CAP Theorem: In distributed systems, you can't have perfect consistency!

How Read Repair Works:

1. Detection: Coordinator requests data from ALL replicas (even if QUORUM only needs 2/3)
2. Comparison: Compares timestamps and data digests from all responses
3. Identification: If timestamps differ, highest timestamp wins (LWW)
4. Repair: Stale replicas are sent the latest version
5. Convergence: All replicas now have consistent data

Two Types:

• Foreground (Blocking): When inconsistency within QUORUM set. Client waits. Adds 10-20ms but guarantees consistency.
• Background (Async): For replicas outside QUORUM. Client doesn't wait. Controlled by read_repair_chance (default 0.1).

Key Benefit: Automatic, continuous healing without manual intervention. Hot data stays consistent, system gradually converges!

FOREGROUND READ REPAIR (Blocking):

When It Triggers:

• Inconsistency detected WITHIN the consistency level
• Example: QUORUM read (needs 2/3), but the 2 responses differ
• MUST repair to satisfy consistency guarantee

Behavior:

• BLOCKING: Client waits for repair completion
• Coordinator sends latest data to stale QUORUM replicas
• Waits for acknowledgment before returning to client
• Adds 10-20ms to read latency
• ALWAYS runs (cannot be disabled for QUORUM+)

Example:

RF=3, QUORUM read (needs 2 responses)
Replica A: data=v2, ts=1000
Replica B: data=v1, ts=900  ← MISMATCH!

Problem: Can't return yet - only 1/2 QUORUM agrees!
Solution: Send v2 → Replica B, wait for ACK
Now 2/2 QUORUM has v2 → return to client ✓
Latency: +15ms (worth it for consistency!)

BACKGROUND READ REPAIR (Async):

When It Triggers:

• Inconsistency detected OUTSIDE the consistency level
• Example: QUORUM consistent (2/3 agree), but 3rd replica is stale
• Consistency already satisfied, want eventual convergence

Behavior:

• ASYNC: Client doesn't wait for repair
• Happens after response sent to client
• Controlled by read_repair_chance (0.0 to 1.0)
• Default: 0.1 (10% of reads)
• Zero latency impact to client

Example:

RF=3, QUORUM read
Replica A: data=v2, ts=1000
Replica B: data=v2, ts=1000  ← QUORUM agrees! ✓
Replica C: data=v1, ts=900   ← Stale, outside QUORUM

QUORUM satisfied → return v2 immediately
Background: Send v2 → Replica C asynchronously
Client latency: 0ms extra ✓

Key Differences:

• Trigger: Foreground = within CL, Background = outside CL
• Blocking: Foreground = yes, Background = no
• Latency: Foreground = +10-20ms, Background = 0ms
• Frequency: Foreground = always, Background = configurable
• Purpose: Foreground = guarantee CL, Background = eventual consistency

Why Both Are Needed: Foreground ensures consistency promises are met. Background gradually heals entire cluster. Together they provide Cassandra's tunable consistency model!

How It Works:

The read_repair_chance parameter (0.0 to 1.0) controls the probability that background read repair will trigger on each read:

1. On each read, Cassandra generates random number (0 to 1)
2. If random < read_repair_chance, trigger background repair
3. Background repair = request from ALL replicas, compare, fix stale ones
4. Default: 0.1 (10% of reads trigger repair)

Tuning Strategy:

1. Start with Default (0.1):

• Good balance for most use cases
• Hot data converges within hours
• Minimal performance overhead

2. Increase for Critical Data (0.2 - 0.5):

• When: Inconsistencies have business impact
• Examples: User accounts, financial data, inventory
• Trade-off: Higher network/CPU for faster convergence
• Monitor: Watch for latency P99 increases

3. Decrease for Read-Heavy (0.05 or lower):

• When: High read volume, eventual consistency OK
• Examples: Analytics, caching, logs
• Benefit: Reduced overhead, faster reads
• Important: Need scheduled nodetool repair!

4. Disable for Write-Only (0.0):

• When: Data rarely/never read
• Examples: Time-series ingestion, audit logs
• CRITICAL: MUST run scheduled repair weekly!

Configuration Examples:

-- Critical user data
ALTER TABLE myapp.users 
WITH read_repair_chance = 0.3;

-- General purpose
ALTER TABLE myapp.products 
WITH read_repair_chance = 0.1;

-- Analytics (with scheduled repair!)
ALTER TABLE myapp.analytics 
WITH read_repair_chance = 0.0;

Monitoring:

• ReadRepairRequests: Total repairs triggered
• RepairedBackground: Should ≈ repair_chance * reads
• RepairedBlocking: Should be <1% (healthy cluster)
• Alert: If blocking >5% = cluster health issue!

Key Takeaway: Higher = faster convergence but more overhead. Lower = less overhead but slower convergence. Find the right balance for each table!

COSTS:

1. Latency Impact:

• Foreground: +10-20ms for inconsistent reads (rare: 0.1-2%)
• Background: 0ms to client (async!)
• P50: Negligible (most reads are consistent)
• P99: +10-15ms (when foreground triggers)

2. Network Overhead:

• Digest requests: 16 bytes (vs 1KB full data = 98% savings!)
• At 10% repair rate: +7-12% network traffic
• Repair writes: Only when inconsistency found (rare)
• Typical: 1-5% of repairs actually find inconsistencies

3. CPU & Memory:

• Digest computation: SHA-256 hash (~100-500μs)
• Comparison: Timestamp + hash compare (~10μs)
• Total CPU: +2-5% at 10% repair rate
• Memory: Minimal (digests cached, ~1MB/1000 rows)

4. Throughput:

• Read throughput: -5-10% with repairs
• Write throughput: No direct impact
• Bottleneck: Usually network, not CPU

BENEFITS:

• Automatic consistency healing (no manual work!)
• No manual repair downtime
• Hot data stays consistent
• Distributed across all nodes

ROI Analysis:

Read Repair Cost (continuous, distributed):
- 10% performance overhead
- Automatic, incremental
- Hot data repairs

Manual Repair Cost (periodic, concentrated):
- 100% resource spike during repair
- Hours to days duration
- Cold data only

Best Approach: BOTH!
- Read Repair: Hot data
- Manual Repair: Weekly for cold data
- Result: Complete coverage, minimal impact

Monitoring Metrics:

nodetool tablestats | grep repair

ReadRepairRequests: 1250/sec
RepairedBlocking: 12/sec (0.96% - healthy! ✓)
RepairedBackground: 125/sec (10% - matches config ✓)

Alert Thresholds:
if blocking_repairs > 2%: WARNING
if blocking_repairs > 5%: CRITICAL

Key Takeaway: 10% overhead for automatic healing is excellent ROI vs 100% spike for manual repair. Monitor foreground repair rate - should be <1% in healthy cluster!

WHEN TO DISABLE (read_repair_chance=0.0):

1. Write-Heavy, Read-Rarely Tables:

• Example: Append-only logs, metrics ingestion, IoT data
• Why: Data rarely read, repair overhead wasted
• Alternative: Weekly/monthly scheduled repair

2. Analytics / Warehouse Tables:

• Example: Historical data, batch processing
• Why: Large scans, eventual consistency acceptable
• Alternative: Full repair before batch jobs

3. High-Throughput Tables:

• Example: Caching layer with 100K reads/sec
• Why: Repair overhead impacts P99 SLA
• Alternative: Scheduled repair during low-traffic windows

4. Multi-DC with Expensive Bandwidth:

• Strategy: DC-local repair only (dclocal_read_repair_chance)
• Cross-DC: Set to 0.0, periodic global repair
• Savings: 90% reduction in cross-DC traffic

ALTERNATIVES TO READ REPAIR:

1. Scheduled Nodetool Repair:

# Incremental repair (recommended)
nodetool repair

# Schedule via cron
0 2 * * 0 nodetool repair -pr  # Weekly Sunday 2AM

Benefits:
- Repairs all data (including cold)
- Much faster than full repair
- Must run regularly (weekly)

2. Cassandra Reaper (Automated Tool):

• Automatic scheduling
• Progress tracking
• Throttling (doesn't saturate cluster)
• Cluster-aware scheduling
• Recommended for large clusters!

3. Hybrid Strategy (BEST):

Hot Data (read often):
  read_repair_chance = 0.1-0.2
  → Repairs automatically via reads

Warm Data (read occasionally):
  read_repair_chance = 0.05-0.1
  + Weekly incremental repair

Cold Data (rarely read):
  read_repair_chance = 0.0
  + Monthly full repair

Result: Complete coverage!

CRITICAL WARNING:

• ❌ NEVER disable without alternative repair strategy!
• ❌ Data will diverge over time
• ❌ Tombstones won't propagate (after gc_grace_seconds)
• ❌ Recovery becomes very difficult

Decision Framework:

1. How often is data read? (frequently = keep repair on)
2. How fast must data converge? (quickly = higher rate)
3. What's backup repair plan? (must have one!)
4. What's cost tolerance? (low = lower rate)

Example - Netflix:

• Viewing history: 0.15 (hot, frequently accessed)
• Recommendations: 0.05 + weekly repair (warm)
• Archives: 0.0 + monthly repair (cold)

Key Takeaway: Only disable when you have reliable alternative. Default 0.1 is good for most cases. Never disable without backup plan!