Troubleshooting

Practice Debugging

Fix real-world Cassandra errors and performance issues!

Home › Practice Debugging
Bug #1
Critical Performance
🐌 Query Takes 30+ Seconds to Complete
Your application's search feature is timing out. Users complain that searching for products by price range takes forever. The query works but performance is unacceptable.
Error Message
Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING
Problematic Query
SELECT * FROM products WHERE price >= 50 AND price <= 100 ALLOW FILTERING;
🔍 Symptoms
  • Query works but takes 30+ seconds
  • CPU usage spikes to 100% during query
  • Affects all nodes in cluster
  • Query scans entire products table
💡 Debugging Hint

ALLOW FILTERING scans ALL partitions, causing full table scan. The problem is that 'price' is not part of the PRIMARY KEY, so Cassandra can't efficiently filter. Think about how to redesign the data model to support this query pattern.

✅ Solution & Explanation

🔍 Root Cause

ALLOW FILTERING forces a full table scan across all partitions. With a large products table, this means reading millions of rows and filtering them in memory. This is why the query is slow.

🛠️ Fix: Create Price Range Table

CREATE TABLE products_by_price_range ( price_bucket text, -- '0-50', '50-100', '100-200', etc. product_id uuid, name text, price decimal, PRIMARY KEY (price_bucket, price, product_id) ) WITH CLUSTERING ORDER BY (price ASC); -- Now query is fast: SELECT * FROM products_by_price_range WHERE price_bucket = '50-100';

💡 Alternative: Secondary Index (with caution)

-- Only if low cardinality + small dataset CREATE INDEX ON products (price);

Warning: Secondary indexes have limitations. Better to design proper tables.

🛡️ Prevention Tips
  • Never use ALLOW FILTERING in production queries
  • Design tables based on query patterns (query-first design)
  • Use bucketing for range queries
  • Avoid secondary indexes on high-cardinality columns
Bug #2
Critical Storage
💾 Massive Partition Warning in Logs
Your IoT sensor application is generating warnings about large partitions. Some queries are timing out, and you're seeing node crashes during compaction.
Log Warning
WARN [CompactionExecutor:1] 2024-01-09 - Compacting large partition sensor_readings:sensor_123 (2.5GB) Partitions larger than 100MB are not recommended
Current Schema
CREATE TABLE sensor_readings ( sensor_id text PRIMARY KEY, reading_time timestamp, temperature decimal, humidity decimal ); -- Sensor collects data every second for years! -- Single partition grows unbounded ❌
🔍 Symptoms
  • Compaction warnings for large partitions
  • Read queries timing out
  • Node crashes during compaction
  • Excessive disk usage on some nodes
💡 Debugging Hint

The problem is that sensor_id is the only partition key. All readings for a sensor go to one partition forever. For time-series data, you need to "bucket" data by time to keep partitions bounded. Think: how can you split data by time periods?

✅ Solution & Explanation

🔍 Root Cause

Using only sensor_id as partition key creates unbounded partitions. One sensor collecting data every second creates ~31 million rows per year in a single partition, far exceeding the 100MB recommendation.

🛠️ Fix: Time Bucketing

CREATE TABLE sensor_readings ( sensor_id text, bucket text, -- 'YYYY-MM-DD' or 'YYYY-MM-DD-HH' reading_time timestamp, temperature decimal, humidity decimal, PRIMARY KEY ((sensor_id, bucket), reading_time) ) WITH CLUSTERING ORDER BY (reading_time DESC); -- Partition key: (sensor_id, bucket) -- Each partition = 1 day/hour of data ✅ -- Bounded partition size!

📝 Insert Example

INSERT INTO sensor_readings ( sensor_id, bucket, reading_time, temperature ) VALUES ( 'sensor_123', '2024-01-09', -- Bucket = today toTimestamp(now()), 23.5 );

🔍 Query Pattern

-- Get readings from specific day SELECT * FROM sensor_readings WHERE sensor_id = 'sensor_123' AND bucket = '2024-01-09'; -- Get last 24 hours (query 2 buckets) SELECT * FROM sensor_readings WHERE sensor_id = 'sensor_123' AND bucket IN ('2024-01-09', '2024-01-08') AND reading_time >= ?;
🛡️ Prevention Tips
  • Always use time bucketing for time-series data
  • Keep partitions under 100MB (ideally under 10MB)
  • Choose bucket size based on write rate
  • Use TimeWindowCompactionStrategy (TWCS) for time-series
  • Monitor partition sizes with nodetool tablestats
Bug #3
High Performance
⚰️ Read Queries Degrading Over Time
Your shopping cart feature was fast initially, but read performance has degraded significantly over 3 months. Queries that took 10ms now take 500ms+.
Log Warning
Read 15000 live rows and 120000 tombstone cells for query SELECT * FROM shopping_carts WHERE user_id = ? (see tombstone_warn_threshold and tombstone_failure_threshold)
Application Code Pattern
// User adds item to cart INSERT INTO shopping_carts (...) VALUES (...); // User removes item (creates tombstone!) DELETE FROM shopping_carts WHERE user_id = ? AND item_id = ?; // Users add/remove items frequently → many tombstones // gc_grace_seconds = 10 days (default) // Tombstones not removed for 10 days!
🔍 Symptoms
  • Read performance degrades over time
  • Tombstone warnings in logs
  • High disk I/O during reads
  • Query latency increases linearly with time
💡 Debugging Hint

Every DELETE creates a tombstone marker. Cassandra must read and skip all tombstones during queries. High delete rates + long gc_grace_seconds = tombstone buildup. Consider: do you really need to DELETE, or can you use TTL or a different pattern?

✅ Solution & Explanation

🔍 Root Cause

Frequent DELETEs create tombstones. During reads, Cassandra must scan through all tombstones (120,000!) to find live data (15,000 rows). Tombstones aren't removed until gc_grace_seconds expires AND compaction runs.

🛠️ Fix 1: Use TTL Instead of DELETE

-- Set TTL on cart items (auto-expire in 30 days) INSERT INTO shopping_carts ( user_id, item_id, quantity ) VALUES (?, ?, ?) USING TTL 2592000; -- 30 days -- No explicit DELETE needed! -- Items expire automatically

Why it works: TTL still creates tombstones, but they're more predictable and manageable.

🛠️ Fix 2: Reduce gc_grace_seconds

ALTER TABLE shopping_carts WITH gc_grace_seconds = 86400; -- 1 day instead of 10 -- Run repair more frequently nodetool repair shopping_carts

Trade-off: Must run repairs more frequently to prevent zombie data.

🛠️ Fix 3: Redesign Without Deletes

-- Add 'active' flag instead of deleting CREATE TABLE shopping_carts ( user_id uuid, item_id uuid, quantity int, active boolean, PRIMARY KEY (user_id, item_id) ); -- Mark as inactive instead of deleting UPDATE shopping_carts SET active = false WHERE user_id = ? AND item_id = ?;

🔧 Immediate Fix: Manual Compaction

nodetool compact keyspace_name shopping_carts

Forces compaction to remove eligible tombstones now.

🛡️ Prevention Tips
  • Minimize DELETEs; use TTL when possible
  • Lower gc_grace_seconds for high-churn tables
  • Monitor tombstone warnings
  • Consider redesigning to avoid deletes
  • Run regular compactions on high-delete tables
Bug #4
High Availability
⏰ Write Timeouts During High Load
During peak traffic, your application experiences write timeouts. Operations that usually succeed start failing with timeout errors.
Error Message
com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency QUORUM (2 replica nodes required but only 1 responded)
Client Configuration
// Application settings consistency_level = QUORUM replication_factor = 3 timeout = 2000ms // 2 seconds // During peak: 1000 writes/sec // Some nodes struggling to keep up
🔍 Symptoms
  • Write timeouts during high traffic
  • Works fine during low traffic
  • Some nodes have high write latency
  • Client sees "2 nodes required but only 1 responded"
💡 Debugging Hint

QUORUM requires 2 out of 3 nodes to respond within timeout. If nodes are overloaded or have high GC pauses, they can't respond in time. Check node performance metrics and consider either fixing slow nodes or adjusting consistency level.

✅ Solution & Explanation

🔍 Root Cause

QUORUM (2/3 nodes) must respond within 2 seconds. During high load, one or more nodes can't respond fast enough. Common causes: high GC pauses, overloaded nodes, network issues, or hardware problems.

🔍 Step 1: Diagnose Node Performance

# Check node stats nodetool tablestats nodetool tpstats nodetool gcstats # Look for: # - High pending writes in tpstats # - Long GC pauses in gcstats # - Uneven load distribution

🛠️ Fix 1: Increase Timeout

// If nodes are just slightly slow timeout = 5000; // Increase to 5 seconds

Note: This is a temporary fix. Find root cause!

🛠️ Fix 2: Lower Consistency Level

// For writes that can tolerate eventual consistency consistency_level = ONE; // Only 1 replica needs to respond // Trade-off: faster writes, slightly less consistency // Still durable (written to commit log + memtable)

🛠️ Fix 3: Tune JVM Heap

# If GC pauses are the issue # In cassandra-env.sh: MAX_HEAP_SIZE="8G" HEAP_NEWSIZE="2G" # Enable G1GC for better pause times -XX:+UseG1GC

🛠️ Fix 4: Add More Nodes

If nodes are genuinely overloaded, scale horizontally by adding more nodes to distribute load.

🛡️ Prevention Tips
  • Monitor node performance metrics continuously
  • Set appropriate timeouts based on P99 latency
  • Use appropriate consistency levels per use case
  • Tune JVM for consistent GC pauses
  • Scale cluster before hitting resource limits
Bug #5
Medium Operations
⚖️ Unbalanced Data Distribution
Your 6-node cluster has uneven data distribution. One node has 500GB while others have 100GB. The overloaded node is slow and causing issues.
nodetool status Output
Datacenter: datacenter1 ======================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Owns Host ID UN 10.0.0.1 102 GB 16.2% abc123... UN 10.0.0.2 495 GB 18.8% def456... ← Problem! UN 10.0.0.3 98 GB 16.5% ghi789... UN 10.0.0.4 105 GB 16.1% jkl012... UN 10.0.0.5 101 GB 16.2% mno345... UN 10.0.0.6 99 GB 16.2% pqr678...
🔍 Symptoms
  • One node has 5x more data than others
  • That node is slower for reads/writes
  • Uneven "Owns" percentages
  • Cluster is not utilizing resources evenly
💡 Debugging Hint

Unbalanced clusters usually indicate either: (1) poor partition key choice creating "hot" partitions, or (2) incorrect token assignment. Check if certain partition keys are storing way more data than others, or if initial_token was manually misconfigured.

✅ Solution & Explanation

🔍 Root Cause Analysis

Two main causes: (1) Hot partitions - one or more partition keys have massively more data, or (2) Bad token distribution - tokens weren't properly assigned when nodes joined.

🔍 Diagnose: Check for Hot Partitions

nodetool cfstats keyspace.table_name # Look at: # - Compacted partition maximum bytes # - Partition size percentiles # If max >> average, you have hot partitions

🛠️ Fix 1: If Hot Partitions (Data Model Issue)

-- BAD: One partition gets all data CREATE TABLE global_events ( event_type text PRIMARY KEY, -- Only 1 value: 'click' ❌ event_id timeuuid, data text ); -- GOOD: Distribute with better partition key CREATE TABLE events ( event_date text, -- 'YYYY-MM-DD' event_type text, event_id timeuuid, data text, PRIMARY KEY ((event_date, event_type), event_id) );

🛠️ Fix 2: If Bad Token Distribution

# Rebalance cluster with vnodes (if not already enabled) # In cassandra.yaml: num_tokens: 256 # Default for vnodes # Decommission and rejoin problematic node nodetool decommission # Wait for completion, then restart node # It will rejoin with proper token distribution

🛠️ Fix 3: Run Cleanup After Decommission

# After rebalancing, clean up old data nodetool cleanup
🛡️ Prevention Tips
  • Use vnodes (num_tokens: 256) for automatic balancing
  • Choose partition keys with high cardinality
  • Monitor "nodetool status" regularly
  • Avoid manual token assignment unless necessary
  • Test data distribution in staging first
Previous
Practice Modeling
Advertisement

Responsive Ad