Critical Operation

Removing Nodes

Decommission safely - Never just stop a node!

⚠️ THE GOLDEN RULE

NEVER JUST STOP/KILL A NODE!

❌ WRONG WAY

# DON'T DO THIS! ❌
sudo systemctl stop cassandra
docker stop cassandra-node3
kill -9 

# Result: PERMANENT DATA LOSS! 💀
                                    

✅ RIGHT WAY

# Do this instead! ✅
nodetool decommission  # Streams data first
# Wait for completion (30 min - 6 hours)
# THEN stop node

# Result: Zero data loss! ✅
                                    

🎯 Decommission = Graceful exit with data transfer
💀 Just stopping = Data loss!

🔀 Decommission vs Removenode

Two different commands for two different scenarios!

✅

nodetool decommission

Graceful removal (PREFERRED)

When to use:

✅ Downsizing cluster
✅ Node is healthy and running
✅ Planned removal
✅ Normal operations

How it works:

Streams data to other nodes
Updates token ownership
Leaves cluster gracefully
Zero data loss ✅

🚨

nodetool removenode

Emergency removal (LAST RESORT)

When to use:

🚨 Node is DEAD (hardware failure)
🚨 Can't access node
🚨 Node won't start
🚨 Emergency only!

How it works:

Forces removal from cluster
NO data streaming
Relies on replication (RF)
Data loss if RF=1 ⚠️

Critical Decision Tree

# Can you SSH to the node?
YES  → Use nodetool decommission ✅
NO   → Is the node permanently dead?
        YES → Use nodetool removenode 🚨
        NO  → Fix the node first!

# Is Cassandra running on the node?
YES  → Use nodetool decommission ✅
NO   → Can you start it?
        YES → Start it, then decommission ✅
        NO  → Use removenode 🚨

# Default: Always try decommission first!
                            

✅ Graceful Decommission Process

The right way to remove a node!

Pre-Decommission Checks

# Check cluster health first
nodetool status

# All nodes should be UN (Up and Normal)
# If any DN (Down), fix those first!

# Check replication factor
cqlsh -e "DESCRIBE KEYSPACE my_keyspace"

# Example output:
# replication = {'class': 'NetworkTopologyStrategy',
#                'datacenter1': '3'}

# RF must be >= 2 to safely remove node!
# If RF=1, you WILL lose data!

# Verify you have enough nodes
# Remaining nodes = Current nodes - 1
# Must be >= RF for all keyspaces
                                

Run Decommission

# SSH to the node you want to remove
ssh node-to-remove

# Start decommission process
nodetool decommission

# This command:
# 1. Changes status to LEAVING
# 2. Streams data to other nodes
# 3. Updates gossip
# 4. Exits cluster

# Time: 30 minutes to 6 hours
# Depends on data size!
                                

Monitor Progress

# Watch the logs
tail -f /var/log/cassandra/system.log

# Key log messages:
# "LEAVING: Starting to leave..."
# "Streaming to multiple nodes..."
# "Finished streaming..."
# "Decommissioned"

# Check from another node
# (SSH to a different node)
nodetool status

# During decommission:
UL  192.168.1.15  100 GB  16  33.4%  ghi789
#^^ UL = Up and Leaving

# Check streaming progress
nodetool netstats  # Run on leaving node
                                

Phase 1: Status Change (Instant)

Node enters LEAVING state

Gossip updated across cluster
Status changes from UN to UL
No new writes directed to this node
Reads still served

Phase 2: Data Streaming (30 min - 6 hours)

Transfers all data to other nodes

Streams SSTables to nodes that will own data
Multiple parallel streams
Progress shown in logs and netstats
This is the longest phase!

Phase 3: Leaving Cluster (1-2 min)

Final cleanup and exit

Verifies all data transferred
Updates gossip state
Removes self from cluster
Cassandra process stops automatically

Post-Decommission

# Decommission complete!
# Cassandra has stopped automatically

# Verify node is gone from cluster
# (Run from another node)
nodetool status

# Node should NOT appear in output

# Check schema agreement
nodetool describecluster

# Optional: Clean up the server
# (On the decommissioned node)
sudo rm -rf /var/lib/cassandra/*
sudo apt remove cassandra  # If done with it
                                

Decommission Complete!

✅ All data transferred to other nodes
✅ Zero data loss
✅ Cluster rebalanced automatically
✅ Node safely removed
✅ Can now repurpose/shutdown server

🚨 Emergency Removenode (Dead Node)

When node is permanently dead!

Use Only When

💀 Hardware failure (server won't boot)
💀 Network partition (can't reach node)
💀 Disk failure (can't read data)
💀 Node is permanently gone
⚠️ You have RF >= 2 (or you WILL lose data!)

Get Dead Node's Host ID

# From a LIVE node, check status
nodetool status

# Output shows dead node:
UN  192.168.1.10  100 GB  16  33.3%  abc123
UN  192.168.1.11  100 GB  16  33.3%  def456
DN  192.168.1.15  100 GB  16  33.4%  ghi789
#^^ DN = Down and Normal                    ^^^^^^^^
#                                           Host ID!

# Copy the Host ID: ghi789
                                

Start Removal Process

# From a LIVE node, run removenode
nodetool removenode ghi789

# Alternative: use IP address
nodetool removenode 192.168.1.15

# This starts the removal process
# Time: 30 min - 4 hours
                                

Monitor Removal

# Check removal status
nodetool removenode status

# Output shows progress:
RemovalStatus: Removing token 1234567890...
RemovalStatus: Finished removing token 1234567890

# Watch logs on LIVE nodes
tail -f /var/log/cassandra/system.log

# Look for:
# "Removing node ghi789..."
# "Node ghi789 removed successfully"
                                

Handle Stuck Removal (If Needed)

# If removal gets stuck (no progress >30 min)
nodetool removenode status

# If still stuck, force it
nodetool removenode force

# This immediately removes node from gossip
# Use with caution!

# After force removal, run repair!
nodetool repair -full
                                

After Emergency Removal

CRITICAL: Run repair on all remaining nodes!

# On EACH remaining node, run:
nodetool repair -full

# Why? Because:
# - Dead node may have had unique data
# - Repair ensures consistency
# - Rebuilds missing replicas

# Time: 1-3 hours per node
# Schedule during low traffic!
                            

📊 Verification Steps

Ensure removal was successful!

✅ Check Cluster Status

# Node should not appear
nodetool status

# Should only show remaining nodes
# All should be UN (Up and Normal)

# Check ring distribution
nodetool ring

# Tokens should be redistributed
                                    

✅ Check Schema Agreement

# All nodes should have same schema
nodetool describecluster

# Should show ONE schema version:
Schema versions:
  5a3b2c1d0-abcd-1234-efgh-567890abcdef: [192.168.1.10, 192.168.1.11]

# If multiple versions, wait or restart gossip
                                    

✅ Verify Data Accessibility

# Test queries still work
cqlsh
SELECT count(*) FROM my_keyspace.users;

# Count should match pre-removal

# Test reads and writes
SELECT * FROM my_keyspace.users WHERE id=123;
INSERT INTO my_keyspace.users (id, name) VALUES (999, 'test');

# Both should work!
                                    

✅ Check Load Distribution

# Check data is balanced
nodetool status

# Load column should increase on remaining nodes
# Example before removal (3 nodes):
UN  node1  100 GB  33.3%
UN  node2  100 GB  33.3%
UN  node3  100 GB  33.4%

# After removal (2 nodes):
UN  node1  150 GB  50%
UN  node2  150 GB  50%
                                    

🔧 Troubleshooting

Fix common issues!

❌ Decommission Stuck/Slow

# Check streaming progress
nodetool netstats

# Check network speed
iftop

# Check for errors in logs
grep -i "error\|exception" /var/log/cassandra/system.log

# If truly stuck (no progress >1 hour):
# 1. Check disk space on receiving nodes
# 2. Check network connectivity
# 3. May need to restart Cassandra on receiving nodes
                                    

❌ Can't Start Decommission

# Error: "Cannot decommission node as it's already decommissioning"

# Check if previous decommission is running
ps aux | grep cassandra

# Check logs
tail -100 /var/log/cassandra/system.log

# If truly stuck, may need to restart Cassandra
sudo systemctl restart cassandra

# Wait for node to be UN, then try again
                                    

❌ Node Still Shows After Removal

# Ghost node in nodetool status

# Try assassinate (last resort!)
nodetool assassinate 192.168.1.15

# This forces removal from gossip
# Use only when node is definitely gone!

# Then restart gossip on all nodes
nodetool disablegossip
nodetool enablegossip
                                    

❌ Data Missing After Removal

# Some data not accessible

# Check replication factor
cqlsh -e "DESCRIBE KEYSPACE my_keyspace"

# If RF=1, data MAY be lost!
# If RF>=2, run repair on all nodes
for node in node1 node2 node3; do
  ssh $node "nodetool repair -full"
done

# This should recover data from replicas
                                    

💡 Best Practices

Do it right!

✅

DO

Always decommission first
Check RF >= 2 before removing
Monitor decommission progress
Remove during low traffic
Verify cluster health after
Run repair after removenode
Document the process
Take snapshot before removal

❌

DON'T

Just stop/kill the node
Remove if RF=1 (data loss!)
Interrupt decommission
Remove during peak traffic
Skip verification steps
Use removenode for healthy nodes
Remove multiple nodes at once
Forget to update monitoring

Production Checklist

Complete this checklist:

✅ Pre-removal: Check RF >= 2, all nodes UN, take snapshot
✅ Decommission: Use nodetool decommission (if node is alive)
✅ OR Removenode: Use nodetool removenode (if node is dead)
✅ Monitor: Watch logs, check streaming progress
✅ Verify: Check status, schema, data access, load distribution
✅ Repair: Run repair if used removenode
✅ Update docs: Update inventory, monitoring, runbooks

🎉 You Can Safely Remove Nodes!

Congratulations! You now know how to remove nodes safely!

🎓 What You Learned:

⚠️ Golden rule: NEVER just stop a node!
🔀 Two methods: Decommission (preferred) vs Removenode (emergency)
✅ Graceful decommission: 3-phase process with data streaming
🚨 Emergency removenode: For dead nodes only
📊 Verification: Status, schema, data, load checks
🔧 Troubleshooting: Stuck decommission, ghost nodes, missing data
💡 Best practices: RF >= 2, monitor, verify, repair

💡 Key Takeaways:

Decommission = Graceful - Streams data first
Removenode = Emergency - For dead nodes only
Check RF >= 2 - Or risk data loss!
Monitor progress - Can take hours
Verify after removal - All checks must pass
Run repair - After emergency removal

📋 Quick Reference:

# Graceful removal (node is alive)
ssh node-to-remove
nodetool decommission
# Wait for completion

# Emergency removal (node is dead)
# Get host ID from another node
nodetool status
nodetool removenode 
nodetool repair -full  # On all remaining nodes!

# Verify
nodetool status
nodetool describecluster
                            

⚠️ Remember: NEVER just stop a node!
Always decommission first!

Responsive Ad