Cluster Expansion

Adding Nodes

Scale your cluster safely and efficiently!

📈 Why Add Nodes to Your Cluster?

Growing your cluster = More capacity, better performance!

🎯 Common Reasons to Add Nodes

💾 Disk capacity: Running out of space (>70% full)
📊 Throughput: Need more read/write capacity
⚡ Performance: Latency increasing under load
🛡️ Redundancy: Improve fault tolerance (RF=3 minimum)
🌍 Geographic expansion: New datacenter
📈 Growth planning: Before hitting limits

📊 Real Example: Netflix

Netflix's Cassandra clusters scale constantly:

🎬 Peak traffic: Friday nights, new releases
➕ Scale up: Add 20-30% more nodes ahead of peak
📈 Result: Zero downtime during surges
💰 Cost: Add nodes incrementally vs over-provisioning

⚡ Cassandra makes scaling easy - add nodes anytime!

⏰

When to Add

Scale proactively!

💾 Disk > 70% full
🔥 CPU > 80% sustained
📊 Latency increasing
⚠️ Thread pool pending tasks
📈 Growth projections

❌

Don't Wait For

Too late!

💾 Disk 95% full
🔥 Constant OOM errors
📉 Severe performance degradation
🚨 Production incidents
😰 Customer complaints

📋 Planning Your Node Addition

Think before you scale!

📊

Capacity Planning

How many nodes do you need?

# Calculate nodes needed for disk capacity
current_data = 500GB per node
projected_growth = 1000GB total in 6 months
keep_at = 70% capacity

nodes_needed = projected_growth / (current_data * 0.7)
              = 1000 / (500 * 0.7)
              = 2.86 → 3 nodes

# Calculate for throughput
current_ops = 10,000 ops/sec per node
target_ops = 50,000 ops/sec total
safety_margin = 1.3x

nodes_needed = (target_ops / current_ops) * safety_margin
              = (50000 / 10000) * 1.3
              = 6.5 → 7 nodes
                                

⚖️

Token Allocation

Cassandra handles this automatically (vnodes)

# Modern Cassandra uses vnodes (virtual nodes)
# Default: 16 vnodes per physical node

# In cassandra.yaml:
num_tokens: 16

# Cassandra automatically:
# 1. Assigns random tokens to new node
# 2. Rebalances data across cluster
# 3. Distributes load evenly

# You don't need to manually calculate tokens!
# (Unless using old single-token per node setup)
                                

Timeline Expectations

Adding a node takes time - plan accordingly:

⚙️ Preparation: 15-30 minutes (config, network, etc)
⏱️ Bootstrap: 1-6 hours (depends on data size)
🔄 Streaming: Most time-consuming part
🧹 Cleanup: 30 minutes - 2 hours (per old node)
📊 Total: Plan for 4-12 hours for complete process

🔧 Preparing the New Node

Get everything ready first!

Hardware/VM Setup

# Provision matching hardware
# Same specs as existing nodes:
# - CPU cores
# - RAM (16GB minimum, 32GB+ recommended)
# - Disk (SSD required, NVMe preferred)
# - Network (10Gbps recommended)

# Verify OS matches
lsb_release -a
# Should match existing nodes

# Update system
sudo apt update && sudo apt upgrade -y
                                

Install Cassandra

# Install SAME version as cluster
# Check existing cluster version first:
nodetool version  # On existing node

# Install that exact version
# Example for Cassandra 4.1.3:
echo "deb https://debian.cassandra.apache.org 41x main" | \
  sudo tee /etc/apt/sources.list.d/cassandra.list

sudo apt update
sudo apt install cassandra=4.1.3

# IMPORTANT: Don't start Cassandra yet!
sudo systemctl stop cassandra
                                

Configure cassandra.yaml

# Edit /etc/cassandra/cassandra.yaml
sudo nano /etc/cassandra/cassandra.yaml

# CRITICAL settings to configure:

# 1. Cluster name (MUST match existing cluster)
cluster_name: 'Production Cluster'

# 2. Seeds (use existing seed nodes)
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "192.168.1.10,192.168.1.11"

# 3. Listen address (this node's IP)
listen_address: 192.168.1.15

# 4. RPC address (for client connections)
rpc_address: 192.168.1.15

# 5. Snitch (MUST match cluster)
endpoint_snitch: GossipingPropertyFileSnitch

# 6. Vnodes (default is fine)
num_tokens: 16

# 7. Data directories
data_file_directories:
  - /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
saved_caches_directory: /var/lib/cassandra/saved_caches
                                

Clean Data Directories

# Remove any existing data
sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*
sudo rm -rf /var/lib/cassandra/saved_caches/*

# Verify directories are empty
ls -la /var/lib/cassandra/data/
ls -la /var/lib/cassandra/commitlog/

# Should show empty directories
                                

Configuration Checklist

VERIFY before starting node:

✅ cluster_name matches existing cluster EXACTLY
✅ seeds point to existing seed nodes
✅ listen_address is this node's IP
✅ rpc_address is this node's IP
✅ endpoint_snitch matches cluster
✅ Cassandra version matches cluster
✅ Data directories are empty
❌ If ANY of these are wrong, node won't join correctly!

➕ Adding the Node - Bootstrap Process

Let's do this!

▶️

Start Cassandra

# Start the new node
sudo systemctl start cassandra

# Enable auto-start on boot
sudo systemctl enable cassandra

# Check status
sudo systemctl status cassandra

# Should show "active (running)"
                                

📊

Monitor Bootstrap Progress

# Watch the log file
tail -f /var/log/cassandra/system.log

# Look for these key messages:
# 1. "Starting up..."
# 2. "Joining ring..."
# 3. "JOINING: Starting to bootstrap..."
# 4. "Streaming from multiple nodes..." (data transfer)
# 5. "Bootstrap completed!"
# 6. "Node is now in NORMAL state"

# Check node status
nodetool status

# Output during bootstrap:
UN  192.168.1.10  100 GB  16  33.3%  abc123
UN  192.168.1.11  100 GB  16  33.3%  def456
UJ  192.168.1.15  50 GB   16  33.4%  ghi789
#^^ UJ = Up and Joining (bootstrapping)

# After completion:
UN  192.168.1.15  100 GB  16  33.4%  ghi789
#^^ UN = Up and Normal (ready!)
                                

Phase 1: Joining Ring (1-2 min)

Node discovers cluster via gossip

Connects to seed nodes
Learns cluster topology
Gets token assignments
Status: UJ (Up and Joining)

Phase 2: Streaming Data (1-6 hours)

Node receives data it will own

Downloads data from multiple nodes
Receives SSTables for its token ranges
Progress shown in logs
Can monitor with nodetool netstats

# Check streaming progress
nodetool netstats

# Shows:
# - Which nodes streaming from
# - Progress percentage
# - Data transferred
                                    

Phase 3: Finalizing (5-10 min)

Node prepares for production

Builds indexes
Compacts received data
Updates gossip state
Status changes to UN (Up and Normal)

Bootstrap Complete!

Node is now part of the cluster and serving traffic!

✅ Status shows UN (Up and Normal)
✅ Load column shows data size
✅ Owns approximately equal token %
✅ Accepting read/write requests
🎉 But wait - you're not done yet! Cleanup needed...

✅ Post-Addition Tasks

Critical cleanup steps!

🧹

Run Cleanup on OLD Nodes

CRITICAL: Old nodes still have data they no longer own!

# Why cleanup?
# Old nodes transferred data to new node
# But still have copies on disk!
# Cleanup removes data they no longer own

# Run on EACH existing node (NOT the new one!)
# SSH to each old node and run:
nodetool cleanup

# Or specific keyspace
nodetool cleanup my_keyspace

# Time: 30 min - 2 hours per node
# Monitor progress:
nodetool compactionstats

# Disk space will be freed after cleanup!
                                

⚠️ Don't skip this! Cleanup frees disk space and ensures correct data distribution.

🔄

Run Repair (Optional but Recommended)

# Ensure data consistency
# Run on the NEW node
nodetool repair -full

# Or specific keyspace
nodetool repair -full my_keyspace

# Time: 1-3 hours depending on data size
# This ensures new node has consistent data
                                

📊

Verify Cluster Health

# Check all nodes are UN
nodetool status

# All should show:
# - Status: UN (Up and Normal)
# - Load: Roughly balanced
# - Owns: Approximately equal

# Check schema agreement
nodetool describecluster

# Should show:
# Schema versions: [only 1 schema UUID]

# Verify data accessibility
cqlsh new_node_ip
SELECT count(*) FROM my_keyspace.users;
# Should return expected count
                                

🔢 Adding Multiple Nodes

Scale up faster!

Important Rules

When adding multiple nodes:

⚠️ Add ONE at a time! Wait for each to complete
⚠️ Don't start multiple simultaneously (overwhelms cluster)
⏰ Wait for UN status before starting next
🧹 Run cleanup after ALL nodes added (not between each)
✅ Order doesn't matter (vnodes handle distribution)

📝

Best Practice: Sequential Addition

# Adding 3 new nodes to cluster

# Step 1: Add node 4
# - Configure cassandra.yaml
# - Start Cassandra
# - Wait for UN status (1-6 hours)
nodetool status  # Verify node4 is UN

# Step 2: Add node 5
# - Configure cassandra.yaml
# - Start Cassandra
# - Wait for UN status (1-6 hours)
nodetool status  # Verify node5 is UN

# Step 3: Add node 6
# - Configure cassandra.yaml
# - Start Cassandra
# - Wait for UN status (1-6 hours)
nodetool status  # Verify node6 is UN

# Step 4: Cleanup ALL original nodes
# - Run on nodes 1, 2, 3
# - NOT on new nodes 4, 5, 6
for node in node1 node2 node3; do
  ssh $node "nodetool cleanup"
done

# Total time: 6-24 hours for 3 nodes
                                

🔍 Troubleshooting

Fix common issues!

❌ Node Won't Join Cluster

Check configuration

# Check cluster name
grep "cluster_name" /etc/cassandra/cassandra.yaml

# Check seeds
grep -A 3 "seed_provider" /etc/cassandra/cassandra.yaml

# Check logs for errors
grep -i "error\|exception" /var/log/cassandra/system.log

# Common issues:
# - Wrong cluster_name
# - Can't reach seeds (firewall/network)
# - Version mismatch
                                    

❌ Bootstrap Stuck/Slow

Network or resource issue

# Check streaming progress
nodetool netstats

# Check network speed
iftop  # Install if needed: apt install iftop

# Check disk I/O
iostat -x 5

# Check if streaming is happening
tail -f /var/log/cassandra/system.log | grep "Streaming"

# If truly stuck (no progress for >30 min):
# 1. Stop node
# 2. Clear data directories
# 3. Fix the issue (network, config, etc)
# 4. Restart bootstrap
                                    

❌ Out of Memory During Bootstrap

Increase heap size

# Check current heap
ps aux | grep cassandra | grep Xmx

# Edit jvm options
sudo nano /etc/cassandra/jvm.options

# Increase heap (example for 16GB RAM)
-Xms8G
-Xmx8G

# Restart Cassandra
sudo systemctl restart cassandra
                                    

❌ Schema Disagreement

Nodes have different schemas

# Check schema versions
nodetool describecluster

# If multiple schema versions shown:
# 1. Wait 60 seconds (usually resolves)
# 2. If persists, restart gossip on problematic node
nodetool disablegossip
nodetool enablegossip

# 3. Check again
nodetool describecluster
                                    

💡 Best Practices

Do it right!

✅

DO

Plan ahead (capacity, timing)
Add during low traffic
Match versions exactly
Wait for UN before next node
Run cleanup on old nodes
Monitor bootstrap progress
Verify cluster health
Document the process

❌

DON'T

Add during peak traffic
Start multiple nodes at once
Use different Cassandra versions
Skip cleanup step
Interrupt bootstrap process
Add when disk >90% full
Forget to enable auto-start
Mix single-token and vnodes

Production Checklist

Complete this checklist for each node:

✅ Pre-addition: Verify capacity need, plan timing, prepare hardware
✅ Configuration: Match cluster name, set seeds, configure IPs
✅ Bootstrap: Start node, monitor logs, verify UN status
✅ Cleanup: Run on old nodes (not new node!)
✅ Verification: Check status, schema, data access
✅ Documentation: Update inventory, runbooks

🎉 You Can Scale Your Cluster!

Congratulations! You now know how to add nodes safely!

🎓 What You Learned:

📈 Why add nodes: Capacity, performance, redundancy
📋 Planning: Capacity calculations, timeline expectations
🔧 Preparation: Hardware, installation, configuration
➕ Bootstrap: 3-phase joining process
✅ Post-addition: Cleanup, repair, verification
🔢 Multiple nodes: Sequential addition strategy
🔍 Troubleshooting: Common issues and fixes
💡 Best practices: Production-ready workflow

💡 Key Takeaways:

Add one at a time - Never start multiple simultaneously
Match configuration - cluster_name, seeds, version
Monitor bootstrap - Watch for UN status
Run cleanup - On OLD nodes after addition
Plan for time - 4-12 hours for complete process
Scale proactively - Before hitting 70% capacity

📋 Quick Reference:

# Adding a node workflow
# 1. Configure cassandra.yaml
# 2. Clean data directories
# 3. Start Cassandra
sudo systemctl start cassandra

# 4. Monitor bootstrap
tail -f /var/log/cassandra/system.log
nodetool status  # Wait for UN

# 5. Cleanup old nodes
nodetool cleanup  # On EACH old node

# 6. Verify health
nodetool status
nodetool describecluster
                            

⚡ Cassandra scales elastically - add capacity anytime!

Responsive Ad