GC GRACE SECONDS - The Grace Period

Tombstone Recap

Tombstone = Deletion marker

When you DELETE data:
• Cassandra writes tombstone
• Not actual deletion!
• Marker says "deleted"
• Has timestamp

Purpose:
Prevents zombie data resurrection
in distributed systems

Tombstones spread to all servers!

Compaction

Compaction = Cleanup process

What it does:
• Merges multiple SSTables
• Removes old data versions
• Deletes expired tombstones
• Frees disk space

When it runs:
• Automatically in background
• Or manually triggered

Compaction is the janitor!

Grace Period

Grace Period = Extra time given
Like a payment grace period

Example:
• Bill due: Jan 1
• Grace period: 10 days
• Can pay until: Jan 11
• No penalty during grace!

For tombstones:
Extra time before removal
Ensures all servers see it!

Zombie Data

Zombie = Deleted data comes back!

How it happens:
1. Delete data on Server A
2. Server B offline (missed it)
3. Tombstone removed too early
4. Server B comes back
5. Still has old data
6. Copies back to A!
7. Zombie! Data resurrected!

Grace period prevents this!

Seconds

Second = Unit of time

Conversions:
• 60 seconds = 1 minute
• 3,600 seconds = 1 hour
• 86,400 seconds = 1 day
• 864,000 seconds = 10 days

Why seconds:
Computer precision
Standard time unit

gc_grace_seconds uses seconds!

Default Setting

Default = Pre-configured value

What it means:
• Value set automatically
• You don't need to configure
• Works out-of-the-box
• Can be changed if needed

gc_grace_seconds default:
864,000 seconds (10 days)

Most users keep the default!

Definition

gc_grace_seconds = Tombstone lifespan

Full name:
"Garbage Collection Grace Seconds"

What it controls:
How long tombstones live
before compaction removes them

Configured per table:
Each table can have different value

Default: 864,000 seconds = 10 days

Purpose

Ensures safe tombstone removal

Protection for:
• Offline servers
• Network partitions
• Maintenance windows
• Failed nodes recovering

Guarantees:
All servers see deletion
before tombstone removed

Result: No zombie data!

How It Works

Step-by-step:

1. Tombstone created
Timestamp: T (deletion time)

2. Compaction runs
Current time: C

3. Check age
Age = C - T

4. Decision
If Age > gc_grace_seconds:
→ Remove tombstone ✓
Otherwise: Keep it!

The Reasoning Behind 10 Days

10 days (864,000 seconds) was chosen based on real-world failure scenarios:

✅ Typical downtime scenarios it covers:

1. Planned Maintenance (1-4 hours):
• OS updates: 1-2 hours
• Hardware upgrades: 2-4 hours
• 10 days: Massive overkill ✓✓✓

2. Network Issues (minutes to hours):
• Router failures: 30 minutes
• Switch replacement: 2 hours
• Data center connectivity: 4-8 hours
• 10 days: Very safe ✓✓

3. Hardware Failures (1-3 days):
• Disk failure: 1 day (replace + restore)
• Server failure: 2 days (ship new hardware)
• Power supply: 1-2 days
• 10 days: Comfortable margin ✓

4. Catastrophic Failures (3-7 days):
• Data center outage: 3-5 days
• Multiple simultaneous failures: 5-7 days
• Emergency procurement: 7 days
• 10 days: Just enough ✓

Additional factors:
• Weekend buffer: Friday failure + weekend = Monday response (3 days)
• Holidays: Extended weekend, slower response
• Human error margin: Mistakes in diagnosis/repair
• Multi-region: Cross-region failures take longer

The philosophy: "Better safe than sorry"
10 days provides enough buffer for 99% of real-world failures while being short enough to not cause severe tombstone accumulation.

Trade-off balance:
• Too short (1 day): Risk zombie data
• Too long (30 days): Tombstone accumulation problems
• 10 days: Sweet spot! ✓

Converting Seconds to Days (Easy!)

Default value: 864,000 seconds

Let's break it down step-by-step:

Step 1: Seconds → Minutes
864,000 seconds ÷ 60 = 14,400 minutes
(60 seconds in 1 minute)

Step 2: Minutes → Hours
14,400 minutes ÷ 60 = 240 hours
(60 minutes in 1 hour)

Step 3: Hours → Days
240 hours ÷ 24 = 10 days ✓
(24 hours in 1 day)

Quick formula:
Days = Seconds ÷ 86,400
(86,400 = seconds in 1 day)

Examples:
• 86,400 seconds = 1 day
• 259,200 seconds = 3 days
• 864,000 seconds = 10 days (default)
• 2,592,000 seconds = 30 days

Remember: 86,400 is your magic number!
Multiply days by 86,400 to get seconds.

gc_grace = 0 seconds

Worst case! NEVER use in production!

What happens:
• Tombstone removed immediately
• Any server offline = risk
• Even 5 minutes down = zombie!

When to use (rare):
• Single-node testing only
• Never run repair
• Throwaway data

Production: DON'T DO THIS!

gc_grace = 1 day (86,400)

Risky but sometimes acceptable

Risk:
• Server down >24 hours = zombie
• Weekend failures problematic
• Hardware delivery takes days

Only if:
• Extremely stable cluster
• 24/7 monitoring
• Rapid response team
• Frequent repairs (every 12h)

Requires operational excellence!

Real Failure Statistics

Actual downtime data:

Planned maintenance:
• 95%: < 4 hours ✓
• 5%: 4-24 hours

Hardware failures:
• 70%: < 1 day
• 25%: 1-3 days
• 5%: 3-7 days ⚠️

If gc_grace = 1 day:
30% of hardware failures = zombie!

10 days covers 95% of failures!

Read Performance

Tombstones slow every query!

Example query:
SELECT * FROM sensors LIMIT 1000

With 10-day grace:
• Scan 5,000 tombstones
• Find 1,000 live rows
• Time: 50ms

With 90-day grace:
• Scan 500,000 tombstones!
• Find 1,000 live rows
• Time: 800ms (16x slower!)

More tombstones = linear slowdown!

Disk Space

Tombstones waste disk space!

Calculation:
• 100M tombstones
• ~20 bytes each
• Total: 2GB wasted!

Impact:
• Disk fills up
• Can't add new data
• Emergency expansion needed
• Expensive!

Shorter grace = more free space!

Compaction Load

More tombstones = slower compaction

With 10-day grace:
• Process 50GB data
• Compaction: 30 minutes

With 90-day grace:
• Process 50GB data + 30GB tombstones
• Compaction: 1.5 hours (3x!)

Vicious cycle:
Slow compaction → tombstones accumulate faster → even slower!

Configuration Commands

-- View current gc_grace_seconds
DESC TABLE users;

-- Set gc_grace to 1 day (86,400 seconds)
ALTER TABLE users WITH gc_grace_seconds = 86400;

-- Set gc_grace to 3 days
ALTER TABLE events WITH gc_grace_seconds = 259200;

-- Set gc_grace to 7 days
ALTER TABLE logs WITH gc_grace_seconds = 604800;

-- Keep default 10 days (no action needed)
-- Default: 864000 seconds

Decision Matrix

Keep default (10 days) if:
• Standard workload ✓
• Unsure what to set
• Low delete volume
• Value stability > performance

Reduce to 1-3 days if:
• High-delete workload
• Stable cluster (99.9% uptime)
• 24/7 monitoring
• Frequent repairs

Increase to 14-30 days if:
• Volatile environment
• Slow response times
• Critical data (no zombie risk)

When in doubt: keep default!

By Workload Type

Time-series with TTL:
• gc_grace: 1-2 days
• High tombstone volume
• Fast removal critical

User data (CRUD):
• gc_grace: 10 days (default)
• Occasional deletes
• Safety important

Append-only logs:
• gc_grace: 10 days
• Rarely delete
• Default fine

Session data:
• gc_grace: 3-5 days
• Moderate deletes
• Balanced approach

Monitoring Required

If reducing gc_grace, monitor:

1. Node downtime:
Alert if down > 50% of gc_grace

2. Repair frequency:
Must run at least every gc_grace period

3. Tombstone warnings:
Watch logs for zombie data

4. Query latency:
Improved after tuning? ✓

No monitoring = keep default!

Uber: Aggressive 1-Day Tuning

Challenge: 270 billion time-series rows with 30-day TTL creating massive tombstone accumulation with default 10-day grace period.

Decision: Reduce to 1 day (86,400 seconds)

Configuration:
ALTER TABLE trip_locations WITH gc_grace_seconds = 86400;

Results:
• Tombstone lifespan: 40 days → 31 days (30-day TTL + 1-day grace)
• Tombstone count: 90B → 9B (10x reduction!)
• P99 latency: 800ms → 45ms (18x faster!)
• Disk usage: 80% tombstones → 15%

Requirements met:
✓ 24/7 monitoring with <12h downtime alerts
✓ Repair every 12 hours (vs weekly)
✓ Highly stable cluster (99.95% uptime)
✓ 2 years production: zero zombie incidents

Key lesson: "For stable, well-monitored clusters, 1-day grace is viable and delivers massive performance gains."

Netflix: Conservative 10-Day Default

Philosophy: "Safety first - we can always optimize later."

Approach: Keep 10-day default for all tables

Rationale:
• Multi-region deployment (cross-region delays)
• Occasional multi-day outages happen
• GDPR compliance critical (no zombie resurrections)
• Performance managed through other optimizations

Alternative optimizations:
• Time-bucketed partitions (avoid tombstones entirely)
• Soft delete + async cleanup
• Aggressive compaction tuning
• Partition drops instead of TTL

Results:
• Zero zombie incidents in 5 years ✓
• Performance good through design, not gc_grace tuning
• Peace of mind for operations team

Key lesson: "Don't tune gc_grace as first optimization - design around deletions instead."

Discord: The 90-Day Disaster

Mistake: Engineer set gc_grace to 90 days "to be extra safe."

Table: user_read_status with 30-day TTL
Configuration: gc_grace_seconds = 7776000 (90 days)

Timeline of pain:
• Month 1: No issues yet
• Month 2: Queries slowing (tombstones accumulating)
• Month 3: P99 latency 15ms → 800ms
• Month 4: Crisis - queries timing out, disk 95% full

Peak problem:
• 200 billion tombstones
• 75% of cluster storage = tombstones
• Scanning 50M tombstones per query
• Compaction: 24/7, still can't keep up

Fix applied:
1. Reduced gc_grace: 90 days → 1 day
2. Forced major compaction
3. Redesigned to time-bucketed partitions

Recovery results:
• After compaction: Tombstones 200B → 2M (99.999% reduction!)
• P99 latency: 800ms → 18ms (44x faster!)
• Disk freed: 2TB

Lesson learned: "Longer gc_grace is NOT always safer - it causes different disasters. Understand the trade-offs!"

1. Start with Default

10 days is proven and safe

Don't tune until you:
• Measure actual problem
• Understand trade-offs
• Have monitoring in place

Premature optimization = danger!

Default works for 90% of use cases ✓

2. Monitor First

Before tuning, measure:

• Tombstone warnings in logs
• Query latency trends
• Disk usage growth
• Compaction frequency

Data-driven decisions only!

No monitoring = keep default

3. Tune Per Table

Different tables, different needs

Time-series: 1-2 days
User data: 10 days
Logs: 10 days
Sessions: 3-5 days

One-size does NOT fit all!

Analyze each table separately

4. Pair with Repair

gc_grace and repair go together

Rule: Repair every gc_grace period

If gc_grace = 3 days:
Run repair every 3 days!

This ensures deletions spread
before tombstones removed ✓

5. Never Use 0

gc_grace_seconds = 0 is DANGEROUS

Only acceptable for:
• Single-node testing
• Throwaway data
• Never repair

Production: NEVER!

Even 1 day is better than 0

6. Design > Tuning

Better: Avoid tombstones entirely!

Instead of tuning gc_grace:
• Time-bucketed partitions
• Soft deletes
• Partition drops
• Design without deletes

Architecture > configuration!