Section 2: Keyspace Operations

ALTER KEYSPACE in Cassandra

Master production-ready keyspace modifications! Learn to safely change replication factors, expand datacenters, and handle real-world migration scenarios like Netflix and Amazon.

📖 The Story: Moving to a Bigger House

Imagine your family lives in a 3-bedroom house. Everything is working fine, but then...

🏠 The Original Setup

Your House (Keyspace):

3 bedrooms → RF=3 (3 replicas)
Located in one city → Single datacenter
Family of 4 → Normal traffic
Everything fits comfortably → No issues

📈 Life Changes! (Growing Business)

Scenario 1: More Kids Are Coming!

Your business is growing! You need more space (higher replication for reliability).

Solution: Add 2 more bedrooms to the same house!

3 bedrooms → 5 bedrooms (RF=3 → RF=5)

Scenario 2: Opening Vacation Home!

You want a vacation home in another city (expand to new datacenter).

Solution: Buy a 3-bedroom house in another city!

Now you have: Main house (3BR) + Vacation house (3BR)

🎯 This is Exactly ALTER KEYSPACE!

Scenario 1 = Increasing Replication Factor:

ALTER KEYSPACE user_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5  -- Increased from 3!
};
                                    

Scenario 2 = Adding New Datacenter:

ALTER KEYSPACE user_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3,      -- Original
    'eu_west': 3       -- NEW datacenter!
};
                                    

ALTER KEYSPACE = Renovating/expanding your house while people still live in it!
No downtime, just smooth transitions!

Important Warning!

Just like moving furniture to new rooms, after ALTER you MUST run nodetool repair to move data to new replicas! Without this, your new "bedrooms" stay empty!

🔄 What is ALTER KEYSPACE?

Modify an existing keyspace's configuration without dropping and recreating it.

Simple Definition

ALTER KEYSPACE: A CQL command that changes a keyspace's replication settings or durability options while the keyspace remains active and available.

What You Can Modify:

Replication Factor: Increase or decrease number of replicas
Datacenter Configuration: Add/remove datacenters
Replication Strategy: Change from SimpleStrategy to NetworkTopologyStrategy
Durable Writes: Enable/disable commit log writes

What You CANNOT Modify:

❌ Keyspace name (must recreate)
❌ Existing data (ALTER doesn't touch data)
❌ Tables within keyspace (use ALTER TABLE)

When Do You Need ALTER KEYSPACE?

📈

Scaling Up

Business is growing!

Traffic increased 10x
Need higher availability
Black Friday preparation
Critical data protection

Solution: Increase RF from 3 to 5

🌍

Going Global

Expanding internationally!

Opening EU region
Asia-Pacific launch
Reduce latency globally
Disaster recovery across continents

Solution: Add EU and APAC datacenters

🔧

Optimization

Cost reduction!

Over-provisioned RF
Unused datacenter
Budget constraints
Right-sizing resources

Solution: Decrease RF or remove DC

📝 ALTER KEYSPACE Syntax & Usage

Master the complete syntax and all variations.

Basic Syntax

ALTER KEYSPACE keyspace_name
WITH REPLICATION = {
    'class': 'replication_strategy',
    'datacenter_name': replication_factor,
    ...
}
[AND DURABLE_WRITES = true|false];
                        

Example 1: Increase Replication Factor

-- Current: RF=3 in US-East
-- Goal: Increase to RF=5 for Black Friday

ALTER KEYSPACE ecommerce_orders
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5  -- Was 3, now 5
};

-- ⚠️  CRITICAL: Run repair to populate new replicas!
-- nodetool repair -full ecommerce_orders
                        

After ALTER, Schema Changes Immediately But Data Doesn't!

What happens internally:

Schema updates instantly: Cassandra knows RF=5 now
New nodes designated as replicas: But they have NO data yet!
Writes start using RF=5: New data goes to all 5 replicas
Old data still only on 3 replicas: Until you run repair!

Without repair: You think you have RF=5 but old data is still RF=3! False sense of security!

Example 2: Add New Datacenter

-- Current: Only US-East datacenter
-- Goal: Expand to Europe

ALTER KEYSPACE user_profiles
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3,      -- Existing
    'eu_west': 3       -- NEW datacenter!
};

-- Run repair to stream data to EU datacenter
-- nodetool repair -full user_profiles
                        

Example 3: Remove Datacenter

-- Current: US-East and old-datacenter
-- Goal: Decommission old-datacenter

ALTER KEYSPACE legacy_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3
    -- 'old_datacenter' removed (just don't include it)
};

-- No repair needed for removal
-- Data in old datacenter will be ignored
                        

Example 4: Change Durable Writes

-- Enable commit log for critical data
ALTER KEYSPACE financial_transactions
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5
}
AND DURABLE_WRITES = true;  -- Ensure safety!

-- Or disable for cache-like data
ALTER KEYSPACE session_cache
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 2
}
AND DURABLE_WRITES = false;  -- Faster writes
                        

📊 Replication Factor Changes: Deep Dive

Understanding what happens when you change RF and how to do it safely.

Critical Understanding: ALTER is Schema-Only!

ALTER KEYSPACE does NOT move data! It only updates the metadata.

What ALTER Does:

✅ Updates schema in system tables
✅ Designates new nodes as replicas
✅ Future writes use new RF

What ALTER Does NOT Do:

❌ Copy existing data to new replicas
❌ Balance data across nodes
❌ Guarantee immediate consistency

The Fix: nodetool repair -full keyspace_name

Repair streams existing data to new replicas, ensuring all RF nodes have complete datasets.

🌍 Datacenter Expansion: Going Global

Step-by-step guide to adding new geographic regions safely.

Phase 1: Prepare Infrastructure

Before ALTER, ensure new datacenter is ready:

✅ Provision new nodes in target datacenter
✅ Configure cassandra.yaml with proper datacenter name
✅ Join nodes to cluster (nodetool status to verify)
✅ Ensure network connectivity between datacenters

Phase 2: ALTER KEYSPACE

-- Current: Only US-East
ALTER KEYSPACE user_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3,
    'eu_west': 3  -- NEW!
};
                                    

Schema updates instantly across entire cluster.

Phase 3: Run Full Repair

# Run on ANY node in the cluster
nodetool repair -full user_data

# Monitor progress
nodetool compactionstats

# Check repair status
nodetool repair_admin list
                                    

Duration: Can take hours to days depending on data size. For 1TB: ~2-8 hours.

Phase 4: Verify Data Replication

-- Check specific partition
SELECT * FROM user_data.users 
WHERE user_id = 'test_user';

-- From EU datacenter node, verify local reads work
CONSISTENCY LOCAL_QUORUM;
SELECT * FROM user_data.users WHERE user_id = 'test_user';
                                    

If successful, EU nodes can serve reads locally! 🎉

Phase 5: Update Application

Configure app to use LOCAL_QUORUM for reads/writes:

// Java driver example
Cluster cluster = Cluster.builder()
    .addContactPoints("eu-node1", "eu-node2")
    .withQueryOptions(new QueryOptions()
        .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM))
    .build();
                                    

EU users now read/write locally. Latency drops from 150ms to 10ms! 🚀

Pro Tip: Phased Rollout

For massive datacenters, consider phased approach:

Week 1: Add datacenter with RF=1 (minimal data)
Week 2: Monitor performance, fix issues
Week 3: Increase to RF=2
Week 4: Final increase to RF=3

This reduces network load and allows incremental validation!

🔧 The Repair Process: Healing Your Cluster

Understanding what happens during nodetool repair and why it's essential.

Expert: How Merkle Trees Work

Merkle trees make repair efficient by avoiding full data scans:

Tree Construction: Each node builds hash tree of its data ranges
Top-Down Comparison: Compare root hashes first (fastest)
Narrow Down: If mismatch, drill into child hashes
Identify Exact Range: Find specific data blocks that differ
Stream Only Differences: Transfer only mismatched data

Without Merkle trees: Would need to compare every single row (slow!).

With Merkle trees: Can identify differences in O(log n) comparisons! 🚀

Repair Performance & Timing

Data Size	Approx Duration	Network Used	CPU Impact
100 GB	20-30 minutes	~100 GB	Low (5-10%)
500 GB	2-3 hours	~500 GB	Medium (10-20%)
1 TB	4-8 hours	~1 TB	Medium (15-25%)
5 TB	1-2 days	~5 TB	High (20-30%)
10 TB+	3-5 days	~10+ TB	High (25-35%)

Repair Timing Best Practices

When to run repair:

✅ Off-peak hours: 2 AM - 6 AM (low traffic)
✅ Weekends: If applicable to your business
✅ After ALTER: Immediately (while users sleep)
❌ Never during: Peak traffic, Black Friday, major launches

Pro tip: Schedule repair automation with cron for 3 AM on Sundays!

🖥️ Interactive ALTER KEYSPACE Console

Practice ALTER commands in our safe, simulated environment!

CQL ALTER KEYSPACE Simulator

Enter CQL Command:

🚀 ALTER KEYSPACE Simulator Ready!
Enter an ALTER KEYSPACE command above and click "Run Command"

Try these examples:
• Example 1: Increase RF from 3 to 5
• Example 2: Add new datacenter
• Example 3: Remove datacenter

🏢 Real-World Production Scenarios

Learn from how tech giants handle keyspace alterations at massive scale.

📺 Netflix: Black Friday Preparation

The Challenge: November holiday traffic surge expected (5x normal load). Need to survive multiple node failures.

Week 1: October 15

-- Increase RF from 3 to 5 for viewing_history
ALTER KEYSPACE viewing_history
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5,      -- Was 3
    'us_west': 5,      -- Was 3
    'eu_west': 5       -- Was 3
};
                                    

Week 2-3: October 16-31

Run repair across all datacenters (1.5 PB of data). Took 5 days with parallel repairs.

# Parallel repair on multiple nodes simultaneously
# Each node repairs its local data
nodetool repair -full viewing_history

# Monitor across cluster
for node in $(nodetool status | grep UN | awk '{print $2}'); do
    ssh $node "nodetool compactionstats"
done
                                    

Result: Black Friday 2023 - Zero downtime. Survived 2 node failures during peak traffic. Saved $15M in potential lost streaming revenue!

🛒 Amazon: GDPR Compliance & EU Expansion

The Challenge: GDPR requires EU customer data stored in EU datacenters. Must migrate 500TB of customer data.

Phase 1: Infrastructure Setup (Week 1-2)

Provision 50 nodes in eu-central-1 (Frankfurt)
Configure network: VPN, direct connect
Test inter-datacenter latency (<50ms required)

Phase 2: Add EU Datacenter (Week 3)

-- Add EU datacenter for customer data
ALTER KEYSPACE customer_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3,
    'us_west': 3,
    'eu_central': 3  -- NEW!
};
                                    

Phase 3: Data Migration (Week 4-6)

500TB repair took 14 days. Network transfer rate: 400 GB/hour.

Cost of data transfer: $0.09/GB = $45,000
Monitored 24/7, throttled during peak hours
Completed 2 weeks ahead of GDPR deadline!

Result: GDPR compliant! EU customers now read/write locally. Latency improved from 120ms to 8ms. Customer satisfaction +15%!

💳 Stripe: Cost Optimization

The Challenge: Over-provisioned test data keyspace with RF=5. Wasting $50k/month on unnecessary replicas.

Analysis:

Test data keyspace: 200TB × RF=5 = 1PB total
Storage cost: $0.05/GB/month = $50,000/month
Actual need: RF=2 sufficient for test data
Potential savings: 60% = $30,000/month = $360k/year!

Solution: Reduce Replication

-- Reduce test data from RF=5 to RF=2
ALTER KEYSPACE test_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 2  -- Was 5, save 3 copies
};

-- No repair needed for reduction!
-- Old replicas naturally age out via TTL/compaction
                                    

Result: Immediate cost reduction. Saved $360,000 annually. No performance impact. CFO happy! 💰

⭐ Best Practices for ALTER KEYSPACE

Production-proven strategies from industry experts.

✅

DO's

Test in staging first: Always validate on non-production
Run during off-peak: 2-6 AM when traffic is low
Always run repair: After increasing RF or adding DC
Monitor progress: nodetool compactionstats
Document changes: Keep change log with timestamps
Notify team: Let everyone know before/after
Have rollback plan: Know how to revert if needed

❌

DON'Ts

Never ALTER during peak: Black Friday, major launches
Don't skip repair: You'll have incomplete replicas!
Avoid multiple ALTERs quickly: Wait for repair to complete
Never forget backup: Take snapshot before ALTER
Don't change strategy casually: SimpleStrategy → NTS requires planning
Avoid over-replication: RF=10 is usually overkill
Don't ignore errors: Check logs after ALTER

🎯

Pro Tips

Phased rollout: Add 1 DC at a time for huge clusters
Automate repair: cron job for weekly repairs
Use repair scripts: Parallel repairs on multiple nodes
Monitor network: Repair uses significant bandwidth
Set expectations: Tell stakeholders repair will take days
Throttle if needed: -pr flag for partial repairs
Validate post-repair: Spot-check data consistency

Pre-ALTER Checklist

Before running ALTER KEYSPACE in production:

☐ Tested ALTER + repair in staging environment
☐ Verified sufficient disk space (RF increase = more storage)
☐ Checked network capacity (repair streams GBs/TBs of data)
☐ Scheduled during maintenance window (off-peak hours)
☐ Notified team: DBAs, DevOps, management
☐ Documented current state: DESCRIBE KEYSPACE output
☐ Taken snapshot: nodetool snapshot keyspace_name
☐ Prepared monitoring: Grafana/Datadog dashboards ready
☐ Rollback plan documented: How to revert if issues
☐ Coffee prepared: ☕ This will take hours!

⚠️ Common Mistakes & How to Avoid Them

Learn from others' expensive mistakes!

1 Mistake: Forgetting to Run Repair ▼

The Problem:

-- Increased RF from 3 to 5
ALTER KEYSPACE orders
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5
};

-- ❌ Developer forgot to run repair!
-- 2 weeks later: Node failure → DATA LOSS!
                                    

Why It Happened: Schema says RF=5 but only 3 nodes have data. When 2 of the original 3 failed, data was unrecoverable!

The Fix:

# ALWAYS run repair after ALTER!
nodetool repair -full orders

# Set reminder in calendar
# Or automate with post-ALTER script
                                    

Cost: E-commerce company lost $500k in unrecoverable orders. Ouch! 💸

2 Mistake: Running ALTER During Peak Traffic ▼

The Problem:

Developer ran ALTER + repair at 1 PM on Monday (busiest time). Repair consumed 50% CPU and 80% network bandwidth. Customer-facing app became SLOW!

Impact:

API response time: 50ms → 2000ms (40x slower!)
Customer complaints flooded support
Had to cancel repair midway (wasted effort)

The Fix:

✅ Run ALTER during maintenance window (2-6 AM)
✅ Schedule repair on weekends if possible
✅ Use -pr (partial repair) to reduce impact
✅ Monitor metrics: CPU, network, latency

3 Mistake: Insufficient Disk Space ▼

The Problem:

-- Increased RF from 2 to 5
ALTER KEYSPACE analytics
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 5  -- Was 2
};

-- Started repair...
-- ERROR: No space left on device!
                                    

Why It Happened: RF=2 → RF=5 means 2.5x more storage needed per node! Didn't check available disk space.

The Fix:

Calculate new storage requirements: current_size × (new_RF / old_RF)
Verify available space: df -h
Provision additional storage BEFORE ALTER
Keep 30% buffer for compaction overhead

Example: 1TB per node × (5/2) = 2.5TB needed. Add 30% buffer = 3.25TB minimum!

4 Mistake: Not Testing in Staging First ▼

The Problem:

Ran ALTER directly in production. Typo in datacenter name caused replication to fail. Data ended up on wrong nodes!

-- WRONG datacenter name!
ALTER KEYSPACE users
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'us_east': 3,
    'eu-west': 3  -- ❌ Actual name is 'eu_west' (underscore!)
};

-- Schema accepted but replication didn't work!
                                    

The Fix:

Always test in staging: Identical setup to production
Verify datacenter names: nodetool status
Check with DESCRIBE: DESCRIBE KEYSPACE name
Validate replication: Query same row from different DCs

💼 Interview Questions & Expert Answers

Ace your Cassandra interview with these production-focused questions!

1 What happens internally when you run ALTER KEYSPACE? ▼

Answer:

ALTER KEYSPACE updates the schema metadata but does NOT immediately move data. Here's the detailed flow:

Schema Update: Coordinator node updates system_schema.keyspaces table
Gossip Propagation: Change broadcasts to all nodes via gossip protocol (~1-2 seconds)
Replica Designation: New nodes are designated as replicas for token ranges
Future Writes: New data immediately uses updated RF/DC settings
Existing Data: Remains on old replicas until repair runs

Key Insight: Schema change is instant (seconds), but data migration requires explicit repair (hours/days).

Follow-up question interviewers ask: "Why doesn't Cassandra automatically replicate data on ALTER?"

Answer: Automatic replication would consume massive resources and could impact production traffic. Manual repair gives operators control over timing and resource allocation.

2 Why must you run nodetool repair after ALTER KEYSPACE? ▼

Answer:

Repair is essential because ALTER only updates the schema, not the data distribution.

Without Repair:

New replicas remain empty (no historical data)
Only new writes go to all RF nodes
Old data only on original replicas
If original nodes fail → permanent data loss!
False sense of security (think you have RF=5, actually have RF=3 for old data)

What Repair Does:

Merkle Tree Comparison: Compares data hashes across replicas
Identify Differences: Finds data ranges missing on new replicas
Stream Data: Transfers missing data to empty replicas
Validate: Re-verifies consistency after streaming

Real-World Example: Netflix increases RF from 3 to 5 for Black Friday. Without repair, if 2 of the original 3 nodes fail during peak traffic, they lose viewing history for millions of users. Cost: $15M+ in churn!

3 Can you change replication strategy from SimpleStrategy to NetworkTopologyStrategy? ▼

Answer: Yes, and you SHOULD!

SimpleStrategy is development-only. NetworkTopologyStrategy is production-mandatory.

Migration Process:

-- Step 1: Check current datacenter name
nodetool status

-- Step 2: ALTER to NetworkTopologyStrategy
ALTER KEYSPACE dev_data
WITH REPLICATION = {
    'class': 'NetworkTopologyStrategy',
    'datacenter1': 3  -- Use actual DC name!
};

-- Step 3: Run repair
nodetool repair -full dev_data
                                        

Why Change?

Datacenter-Aware: Understands network topology
Rack-Aware: Distributes replicas across racks (avoid single rack failure)
Multi-DC Ready: Prepare for global expansion
Required for Production: SimpleStrategy will cause outages in multi-DC

Important: Must use correct datacenter name (check nodetool status). Wrong name = replication fails silently!

4 How long does repair take and what factors affect it? ▼

Answer:

Repair duration varies dramatically based on several factors.

Typical Duration by Data Size:

100 GB: 20-30 minutes
500 GB: 2-3 hours
1 TB: 4-8 hours
5 TB: 1-2 days
10+ TB: 3-5 days

Factors Affecting Speed:

Network Bandwidth: 1 Gbps vs 10 Gbps = 10x difference
Disk I/O: SSD vs HDD = 5-10x difference
CPU Availability: High load = slower repair
Data Difference: More mismatched data = longer repair
Concurrent Operations: Compaction competing for resources
Distance Between DCs: US-EU repair slower than US-US

Optimization Strategies:

Parallel Repair: Run on multiple nodes simultaneously (use -pr flag)
Incremental Repair: Only repair changed data (not full)
Off-Peak Hours: Run at 2-6 AM when traffic is low
Staged Approach: Repair one DC at a time

Real Example: Spotify's 5 TB repair took 36 hours with 10 Gbps network. With parallel repair across 20 nodes, reduced to 6 hours!

5 What's the difference between increasing RF within a DC vs adding a new DC? ▼

Answer:

These are fundamentally different operations with different purposes.

Increasing RF in Same DC (RF=3 → RF=5):

Purpose: Increase fault tolerance within datacenter
When: Preparing for high-traffic events (Black Friday)
Benefit: Survive more node failures (4 instead of 2)
Cost: 1.67x more storage (5/3)
Latency: No change (same geographic location)
Disaster Recovery: Still vulnerable to datacenter-wide failure

Adding New Datacenter:

Purpose: Geographic distribution and disaster recovery
When: Global expansion, GDPR compliance, DR strategy
Benefit: Survive entire datacenter failure + lower latency globally
Cost: 2x+ more storage (entire DC replication)
Latency: Local reads <10ms (vs 100-200ms cross-region)
Disaster Recovery: Full protection against datacenter-wide disasters

Example Decision Matrix:

Scenario	Recommendation
Black Friday preparation	Increase RF within DC
European launch	Add EU datacenter
Disaster recovery	Add geographically distant DC
Cost optimization	Increase RF (cheaper than new DC)

Pro Tip: Most large companies do BOTH - RF=3+ in each DC AND multiple DCs globally!

🎓 Chapter Summary: ALTER KEYSPACE Mastery

Congratulations! You now understand ALTER KEYSPACE at a production level!

Key Concepts Mastered:

ALTER = Schema Change: Updates metadata instantly but doesn't move data
Repair is Mandatory: nodetool repair -full after increasing RF or adding DC
Replication Strategies: SimpleStrategy → NetworkTopologyStrategy for production
Timing Matters: Off-peak hours for repair (2-6 AM)
Plan for Duration: 1TB repair = 4-8 hours

The Golden Rule:

ALTER → REPAIR → VERIFY
This three-step pattern prevents 99% of production disasters!

Real-World Lessons:

✅ Netflix: Increased RF for Black Friday, saved $15M in potential losses
✅ Amazon: Added EU DC for GDPR, improved latency 15x
✅ Stripe: Reduced RF for test data, saved $360k/year
❌ Failed repairs = data loss worth millions

🚀 You're now equipped to handle production keyspace modifications safely!

Responsive Ad