ROW CACHE - Lightning-Fast Memory

Complete Answer:

Row cache is an in-memory cache in Cassandra that stores complete rows in the JVM heap to dramatically speed up read queries by avoiding disk I/O.

What It Is:

• Location: JVM heap memory (same memory space as Cassandra process)
• Stores: Complete deserialized rows (not just keys)
• Speed: 0.5ms access (vs 30ms disk read = 60x faster)
• Size: Configurable per-table (typically 100MB-2GB)
• Eviction: LRU (Least Recently Used) - oldest unused rows removed when full
• Persistence: Can be saved to disk between restarts

How It Works:

1. Query arrives: SELECT * FROM users WHERE id='user123'
2. Check cache: Is this row already cached? (0.5ms lookup)
3. Cache HIT: Return immediately, skip all disk I/O
4. Cache MISS: Read from memtable/SSTables (30ms), then store in cache for next time

When to Use Row Cache (Perfect Scenarios):

1. Read-Heavy Tables:

• Read:write ratio > 10:1
• Example: User profiles, product catalogs
• Why: Writes invalidate cache entries, so write-heavy defeats purpose

2. Small Rows:

• Row size < 1KB ideal
• Example: User sessions (800 bytes), metadata
• Why: 2GB cache holds 2.5 million 800-byte rows vs only 20,000 100KB rows
• More rows cached = higher hit rate

3. Hot Data Access Pattern:

• Same rows accessed repeatedly
• Example: Celebrity profiles on Instagram, popular products
• Why: High hit rate (80%+) is essential for cache to be worthwhile

4. Latency-Sensitive Applications:

• Need sub-5ms P99 latency
• Example: Real-time dashboards, gaming leaderboards
• Why: Cache makes 95%+ of queries sub-millisecond

When NOT to Use Row Cache (Avoid These):

1. Large Rows:

• Row size > 10KB
• Example: Images, documents, large JSON blobs
• Problem: Cache fills with few rows, low hit rate

2. Write-Heavy Tables:

• Write:read ratio > 1:5
• Example: Time-series data, logs
• Problem: Every write invalidates cache entry, constant churn

3. Uniform Access Pattern:

• Every row accessed equally (no "hot" data)
• Example: Analytical scans, full table queries
• Problem: Low hit rate, cache constantly thrashing

4. Time-Series Data:

• Data accessed once then never again
• Example: Logs, metrics with TTL
• Problem: Cache misses on every new data point

Real-World Example (Instagram):

• Scenario: User profiles table, 1 billion users
• Row size: 400 bytes per profile
• Access pattern: Celebrity profiles accessed millions of times/day
• Configuration: 2GB row cache per node
• Capacity: 5 million profiles cached per node
• Hit rate: 92% (hot users always cached)
• Impact: P99 latency 3ms (was 45ms without cache)
• Benefit: 15x faster, avoided 200 additional servers

Trade-offs to Consider:

• Memory cost: Uses JVM heap (increases GC pressure)
• GC impact: Larger cache = longer GC pauses
• Sweet spot: 10-25% of heap size
• Monitor: Hit rate must be 80%+ to justify memory cost

Key Insight: Row cache is NOT a universal optimization. It's a specialized tool for specific workloads: read-heavy, small rows, hot data. When used correctly (90%+ hit rate), it's transformative - 60x faster reads. When misused (low hit rate, large rows), it wastes RAM and increases GC pauses. Always test and measure before enabling in production!

Complete Answer:

Cache hit and miss represent the two possible outcomes when checking if data exists in cache. Hit rate is the most critical cache metric because it directly determines your read performance and cost efficiency.

Cache HIT - The Good Path:

• Definition: Requested data found in cache
• Path: Query → Check cache → Found! → Return immediately
• Latency: ~0.5ms (memory access)
• Disk I/O: Zero (huge win!)
• Example: Query user123, it's cached from previous query

Cache MISS - The Slow Path:

• Definition: Requested data NOT in cache
• Path: Query → Check cache → Not found → Check memtable → Check SSTables on disk → Load & decompress → Store in cache → Return
• Latency: ~30ms (disk access = 60x slower)
• Disk I/O: Full read path executed
• Example: First time querying user456

Hit Rate Formula:

Hit Rate = (Cache Hits / Total Queries) × 100
Example: 900 hits, 100 misses = 900/(900+100) × 100 = 90%

Why Hit Rate Matters - The Math:

Scenario: 1000 queries/second

With 90% hit rate (GOOD):

• 900 queries hit cache @ 0.5ms = 450ms total
• 100 queries miss cache @ 30ms = 3,000ms total
• Total time: 3,450ms
• Average latency: 3.45ms per query ✓

With 50% hit rate (BAD - cache too small):

• 500 queries hit cache @ 0.5ms = 250ms total
• 500 queries miss cache @ 30ms = 15,000ms total
• Total time: 15,250ms
• Average latency: 15.25ms per query ❌

Impact: 90% vs 50% hit rate = 4.4x performance difference!

Real-World Impact Example (Instagram):

Before row cache (0% hit rate):

• Every query hits disk
• P50: 18ms, P99: 45ms
• Disk IOPS maxed out at 50k queries/sec
• Needed 50 database servers

After row cache (92% hit rate):

• 92% skip disk entirely
• P50: 0.8ms, P99: 3.2ms
• Throughput: 500k queries/sec (10x!)
• Needed only 5 servers

Why Each Percentage Point Matters:

• 80% → 85%: 5% more queries skip 30ms disk read = ~1.5ms average latency improvement
• 85% → 90%: Another ~1.5ms improvement
• 90% → 95%: Another ~1.5ms improvement
• Pattern: Linear improvement, every % counts!

Hit Rate Thresholds:

• 95%+: Excellent - cache sized perfectly
• 85-95%: Very good - operating efficiently
• 75-85%: Good - room for improvement
• 60-75%: Mediocre - consider increasing cache size
• <60%: Poor - cache too small OR wrong data cached

Factors Affecting Hit Rate:

1. Cache Size:

• Larger cache = more rows fit = higher hit rate
• But: Larger cache = more GC pressure
• Sweet spot: 10-25% of heap

2. Working Set Size:

• Working set = frequently accessed rows
• If working set fits in cache → high hit rate
• If working set larger than cache → low hit rate

3. Access Pattern:

• Skewed (hot data): High hit rate possible
• Uniform (all data equal): Low hit rate inevitable
• Example: 80/20 rule - 80% of queries hit 20% of data

4. Write Frequency:

• Every write invalidates cached row
• High write rate = constant cache invalidation = low hit rate

Monitoring Hit Rate:

nodetool info | grep -A 3 "Row Cache"

# Output example:
Row Cache              : entries 1250000, size 1.8 GiB, capacity 2 GiB
                         45000000 hits, 5000000 requests, 0.900 recent hit rate
# 90% hit rate = excellent!

Improving Low Hit Rate:

• If <80%: Increase cache size
• Still low: Wrong table for caching (uniform access or large rows)
• Frequent invalidation: Table too write-heavy
• Decision: If can't achieve 80%+, disable cache (wasting RAM)

Cost Implications:

Example: 10,000 queries/sec needed

50% hit rate (inefficient):

• Average 15ms/query
• Need 30 servers @ $500/mo = $15,000/month

90% hit rate (efficient):

• Average 3.5ms/query
• Need 5 servers @ $500/mo = $2,500/month
• Savings: $12,500/month = $150k/year!

Key Insight: Hit rate is THE most important cache metric. The difference between 50% and 90% is the difference between acceptable and terrible performance - literally 4-5x. Monitor it obsessively. If you can't achieve 80%+ hit rate, the cache is wasting resources and should be disabled. When hit rate is high (90%+), cache provides 10-50x performance improvement and massive cost savings!