READ PATH - Query Execution Journey

Complete Answer:

The read path is the journey of a query from client to data and back, involving multiple nodes and optimization layers.

Step-by-Step Flow:

1. Client sends query:

• CQL query: SELECT * FROM users WHERE id=123
• Specifies consistency level (e.g., QUORUM)
• Connects to any Cassandra node

2. Coordinator node receives query:

• The receiving node becomes coordinator
• Parses query and extracts partition key (id=123)
• Hashes partition key to token: hash(123) = 789456123
• Looks up token ring to find replica nodes
• Example: Identifies Node 1, Node 3, Node 8 (RF=3)

3. Coordinator sends parallel requests:

• Sends read request to ALL replica nodes simultaneously
• For QUORUM: needs 2 of 3 responses
• Optimization: Request full data from CL nodes, digest from others
• Parallel execution minimizes latency

4. Each replica checks its caches:

• Row cache: If enabled and hit → return immediately (0.5ms)
• Memtable: Check in-memory recent writes (1ms)
• Key cache: If hit → know SSTable offset, skip Index.db (3ms)
• Bloom filters: Check each SSTable "is key here?"
• SSTables: If no cache hit, read from disk (10-50ms)

5. Replicas return data to coordinator:

• Each replica sends: data + timestamp
• Example: Node 1: {data, ts=1640995200}, Node 3: {data, ts=1640995200}
• Coordinator waits for CL responses (QUORUM = 2)
• Returns as soon as minimum responses received

6. Coordinator compares timestamps:

• Checks all response timestamps
• If mismatch detected: Node 1 (ts=100), Node 3 (ts=95)
• Selects newest data (highest timestamp)
• Triggers read repair for outdated replicas

7. Read repair (background):

• Send newest data to stale replicas
• Asynchronous - doesn't block response
• Repairs inconsistencies transparently

8. Return to client:

• Coordinator sends newest data to client
• Total latency: Determined by slowest required replica
• Example: Cache hit = 2ms, disk read = 20ms

Key Optimizations:

• Parallel queries: All replicas queried simultaneously
• Fast path: Return as soon as CL satisfied
• Caching layers: Row cache → memtable → key cache → OS cache
• Bloom filters: Skip 99% of unnecessary SSTable reads
• Read repair: Self-healing, maintains consistency

Real-World Example (Netflix):

• 1 billion reads/day
• RF=3, CL=LOCAL_QUORUM
• 80% cache hit rate
• P50: 1.5ms, P99: 5ms
• Read repair keeps replicas in sync

Key Insight: The read path balances consistency, latency, and availability through tunable consistency levels, aggressive caching, parallel queries, and automatic read repair. This architecture enables billion-scale reads with sub-10ms latency!

Complete Answer:

The coordinator uses consistent hashing and the token ring to determine which nodes own a partition's data.

The Token Ring Concept:

• Hash space: 2^64 possible token values (0 to 2^64-1)
• Ring structure: Tokens arranged in circular order
• Node ownership: Each node owns range of tokens
• Partitioner: Murmur3Partitioner (default) hashes keys

Step-by-Step Lookup Process:

1. Extract partition key from query:

• Query: SELECT * FROM users WHERE id=123
• Partition key: id=123

2. Hash the partition key:

• Apply Murmur3 hash function
• hash(123) = 7894561230000000000 (example token)
• Deterministic: Same key always produces same token

3. Locate position on ring:

• Token ring: Node1(0-100), Node2(101-200), Node3(201-300)...
• Find which node owns token 7894561230000000000
• Example: Falls in Node 5's range
• Node 5 = Primary replica

4. Determine additional replicas (RF>1):

• RF=3 means 3 total replicas needed
• Walk clockwise around ring from primary
• Next nodes: Node 5 (primary), Node 8, Node 1
• NetworkTopologyStrategy: Considers DC/rack for placement

5. Check network topology:

• Multi-DC: Ensure replicas spread across DCs
• Example RF={US-East:3, EU-West:3} = 6 total replicas
• Rack awareness: Spread across racks for fault tolerance

6. Coordinator sends requests:

• Now knows all replica locations
• For LOCAL_QUORUM: Only query local DC replicas
• Send parallel requests to identified nodes

Concrete Example:

Setup: 9 nodes across 3 DCs, RF={US:3, EU:3, ASIA:3}

Query: SELECT * FROM users WHERE id=456 with CL=LOCAL_QUORUM (in US-East DC)

Process:

• 1. Hash partition key: hash(456) = 3333333333333333333
• 2. Lookup token ring: Falls in Node 2's range (US-East)
• 3. Identify replicas in US-East: Node 2, Node 5, Node 8
• 4. LOCAL_QUORUM needs 2 of 3 US-East nodes
• 5. Coordinator queries all 3, waits for 2 responses
• 6. Ignores EU and ASIA replicas (LOCAL_* consistency level)

Key Advantages of Token Ring:

• No central metadata: Every node knows topology
• Fast lookup: O(log N) to find owning node
• Scalable: Add nodes without rehashing all data
• Fault-tolerant: Multiple replicas ensure availability
• Consistent: Same key always maps to same nodes

Token-Aware Drivers:

• DataStax drivers cache cluster topology
• Client can compute token and find owning node
• Sends query directly to primary replica
• Saves coordinator hop (1-2ms)

Virtual Nodes (vnodes):

• Default: 256 vnodes per physical node
• Each vnode owns small token range
• Better load distribution
• Faster rebalancing when nodes added/removed
• Lookup: Same process, just more granular ranges

Key Insight: The token ring + consistent hashing enables truly distributed architecture. No coordinator election, no leader bottleneck. Every node knows topology and can route to correct replicas instantly. This is how Cassandra scales linearly!

Complete Answer:

Caching is critical because disk I/O is 100-1000x slower than memory access. Cassandra uses multiple cache layers to maximize read performance.

The Speed Hierarchy:

• Memory (RAM): 100 nanoseconds
• SSD: 100 microseconds (1000x slower)
• HDD: 10 milliseconds (100,000x slower!)

Without caching: Every read hits disk = 10-50ms latency. At 1000 reads/sec = system overwhelmed!

Layer 1: Row Cache (Fastest - 0.5ms):

• What: Complete row stored in JVM heap
• Location: On-heap memory
• Size: Configurable per-table (100MB-2GB typical)
• Hit: Return immediately, skip everything else!
• Best for: Read-heavy tables with small rows
• Example: User profiles, session data
• GC impact: Larger cache = longer GC pauses
• Trade-off: Speed vs heap pressure

Layer 2: Memtable (1-2ms):

• What: Recent writes cached in memory
• Always checked: Before SSTables
• Size: 64-256MB typically
• Hit rate: High for recently written data
• Speed: In-memory sorted structure (skip list)

Layer 3: Key Cache (3-5ms):

• What: Maps partition key → SSTable offset
• Location: Off-heap (no GC impact!)
• Benefit: Skip Index.db read (saves 5-10ms)
• Size: 100MB-1GB typical
• Always enabled: Default on
• Hit rate: 80-95% for hot data
• Critical: Huge performance impact

Layer 4: OS Page Cache (5-10ms):

• What: Operating system caches frequently accessed files
• Automatic: Linux kernel manages
• Size: All RAM not used by JVM
• Caches: SSTable data files
• Hit: Much faster than disk read
• Impact: 50-80% of disk reads avoided
• Best practice: Leave 50% RAM for OS cache

Layer 5: Disk Read (10-50ms - Slowest):

• Only if all caches miss
• Steps: Bloom filter → Index.db → Data.db
• SSD: 10-20ms
• HDD: 30-50ms
• Goal: Minimize disk reads!

Real Impact Example (Spotify):

Setup: 64GB RAM nodes, 1000 reads/sec

Configuration:

• Row cache: 2GB (user profiles)
• JVM heap: 16GB
• Key cache: 1GB
• OS page cache: 45GB

Cache Hit Distribution:

• Row cache hit: 60% @ 0.8ms = 600 reads @ 480ms total
• Memtable hit: 15% @ 1.5ms = 150 reads @ 225ms total
• Key cache hit: 20% @ 4ms = 200 reads @ 800ms total
• OS cache hit: 4% @ 8ms = 40 reads @ 320ms total
• Disk read: 1% @ 30ms = 10 reads @ 300ms total

Total: 2125ms for 1000 reads = 2.1ms average ✓

Without Caching (all disk reads):

• 1000 reads @ 30ms = 30,000ms total
• Average: 30ms per read
• 14x slower!

Cache Hit Rate Impact:

• 95% hit rate: Average 2ms
• 80% hit rate: Average 7ms
• 50% hit rate: Average 16ms
• Every 10% improvement: ~2ms faster

Configuration Best Practices:

• Row cache: Enable for read-heavy tables with small rows
• Key cache: Always keep enabled
• JVM heap: ≤50% of total RAM (leave room for OS cache)
• Monitor: Cache hit rates constantly
• Alert: If combined hit rate drops below 75%

Why Multiple Layers:

• Row cache: Fastest but limited size (heap pressure)
• Key cache: Larger size possible (off-heap)
• OS cache: Huge size (free RAM), automatic
• Together: Cover different access patterns
• Fallback: If one misses, next layer tries

Key Insight: Caching is THE most important read optimization. Disk is 1000x slower than memory. With proper caching (95%+ hit rate), Cassandra achieves sub-2ms reads at billion-scale. Without caching, it would be unusable for real-time workloads!