Cluster Management Tool

Master Nodetool

Your Swiss Army knife for Cassandra cluster administration!

Home › Tools › Nodetool Guide

🔧 What is Nodetool?

Nodetool = Your cluster management Swiss Army knife!

It's THE command-line tool for managing and monitoring Cassandra clusters. Think of it as:

  • 🔧 Administrator's toolkit for cluster operations
  • 📊 Monitoring dashboard in your terminal
  • 🔍 Debugging tool for investigating issues
  • 💾 Backup manager for snapshots
  • 🏥 Health checker for cluster status

✨ What You Can Do

  • ✅ Check cluster health: Node status, ring topology
  • ✅ Maintenance tasks: Repair, compact, flush, cleanup
  • ✅ Monitor performance: Stats, metrics, thread pools
  • ✅ Manage nodes: Add, remove, decommission, rebuild
  • ✅ Backup/restore: Snapshots, incremental backups
  • ✅ Tune settings: Cache, compaction, streaming

🎯 When to Use Nodetool

  • 📊 Daily: Check cluster status
  • 🔧 Weekly: Run repairs, check stats
  • 🚨 Troubleshooting: Investigate issues, view logs
  • ➕ Scaling: Add/remove nodes
  • 💾 Backups: Create snapshots
  • ⚡ Performance: Analyze bottlenecks

🎓 Master nodetool = Master Cassandra operations!

Running Nodetool

# Linux/Mac (if in PATH) nodetool status # Full path /opt/cassandra/bin/nodetool status # Docker docker exec cassandra-node1 nodetool status # Remote node nodetool -h 192.168.1.100 status # With JMX authentication nodetool -u jmxuser -pw jmxpassword status

⭐ Essential Commands - Use These Daily

The commands you'll use most often!

🏥 nodetool status

THE most important command! Shows cluster health.

# Check cluster status nodetool status # Output explanation: Datacenter: datacenter1 Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID UN 172.18.0.2 69 KiB 16 33.3% abc123... UN 172.18.0.3 65 KiB 16 33.3% def456... DN 172.18.0.4 73 KiB 16 33.4% ghi789... # Status codes: # UN = Up and Normal ✅ # DN = Down and Normal ❌ # UL = Up and Leaving # UJ = Up and Joining # Specific keyspace nodetool status my_keyspace

ℹ️ nodetool info

Detailed node information

# Get node details nodetool info # Shows: # - ID and generation # - Gossip active: true # - Load: 69.91 KiB # - Heap Memory: 256/1024 MB # - Off Heap Memory # - Data Center / Rack # - Exceptions # - Key Cache / Row Cache

🔍 nodetool describecluster

Cluster overview

# Get cluster information nodetool describecluster # Shows: # - Cluster Name # - Snitch # - DynamicEndPointSnitch # - Partitioner # - Schema versions (should be 1!)

💍 nodetool ring

Token ring distribution

# View token ring nodetool ring # Shows which node owns which token ranges # Useful for debugging data distribution # For specific keyspace nodetool ring my_keyspace

📋 nodetool version

Check Cassandra version

# Get version nodetool version # Output: ReleaseVersion: 4.1.3

🔧 Maintenance Commands

Keep your cluster healthy!

🔄

nodetool repair

CRITICAL: Ensures data consistency across replicas

# Full repair (all keyspaces) nodetool repair # Specific keyspace nodetool repair my_keyspace # Specific table nodetool repair my_keyspace users # Full repair (recommended weekly) nodetool repair -full # Primary range only (faster) nodetool repair -pr # Parallel repair (use with caution) nodetool repair -j 4 # 4 threads

⏱️ Time: Can take minutes to hours depending on data size

📅 Schedule: Weekly for critical data, monthly for non-critical

🗜️

nodetool compact

Merge SSTables to reclaim space and improve performance

# Compact all keyspaces nodetool compact # Specific keyspace nodetool compact my_keyspace # Specific table nodetool compact my_keyspace users # Split output (create multiple SSTables) nodetool compact -s my_keyspace users

⚠️ Warning: Use sparingly! Compaction happens automatically

💡 When to use: After bulk deletes, when disk is low

🧹

nodetool cleanup

Remove data this node no longer owns (after adding nodes)

# Cleanup all keyspaces nodetool cleanup # Specific keyspace nodetool cleanup my_keyspace # Specific table nodetool cleanup my_keyspace users

🎯 Purpose: Reclaim disk space after cluster expansion

📅 When: After adding new nodes to cluster

💾

nodetool flush

Write memtables to disk immediately

# Flush all keyspaces nodetool flush # Specific keyspace nodetool flush my_keyspace # Specific table nodetool flush my_keyspace users

💡 When to use: Before backups, before shutdown, testing

⚡ Fast: Usually completes in seconds

Maintenance Best Practices

  • ✅ Repair: Run weekly on production clusters
  • ⚠️ Compact: Only when necessary (not routinely)
  • ✅ Cleanup: Always after adding nodes
  • ✅ Flush: Before backups and shutdowns
  • ⚠️ I/O Impact: These operations use disk heavily
  • 💡 Scheduling: Run during low-traffic periods

📊 Monitoring Commands

Monitor performance and health!

📊 nodetool tablestats

Table-level statistics

# All keyspaces nodetool tablestats # Specific keyspace nodetool tablestats my_keyspace # Specific table (with human-readable sizes) nodetool tablestats -H my_keyspace.users # Shows: # - Space used # - SSTable count # - Read/Write count # - Read/Write latency # - Pending compactions

🧵 nodetool tpstats

Thread pool statistics

# Thread pool stats nodetool tpstats # Shows for each pool: # - Active threads # - Pending tasks # - Completed tasks # - Blocked threads # Watch for: # - High pending counts (backpressure) # - Many blocked threads (contention)

📈 nodetool proxyhistograms

Latency histograms

# Latency distribution nodetool proxyhistograms # Shows percentiles: # - 50%, 75%, 95%, 98%, 99%, 99.9% # For reads, writes, range queries # Example output: Percentile Read Latency Write Latency 50% 1.2 ms 0.8 ms 95% 5.4 ms 3.2 ms 99% 12.1 ms 8.7 ms

🗄️ nodetool cfstats

Column family (table) statistics - Detailed

# All tables (verbose) nodetool cfstats # Specific keyspace nodetool cfstats my_keyspace # More detailed than tablestats # Includes bloom filter stats, compression, etc

📉 nodetool compactionstats

Active compactions

# Check compaction status nodetool compactionstats # Shows: # - Active compactions # - Progress percentage # - Type (major, minor, cleanup, etc) # - SSTables being compacted # Use with -H for human-readable sizes nodetool compactionstats -H

💾 nodetool getlogginglevels

View current log levels

# Show all log levels nodetool getlogginglevels # Set log level dynamically nodetool setlogginglevel org.apache.cassandra DEBUG # Reset to default nodetool setlogginglevel org.apache.cassandra INFO

🌐 Cluster Operations

Scale and manage your cluster!

➖

nodetool decommission

CRITICAL: Gracefully remove a node

# Decommission THIS node # Run on the node you want to remove nodetool decommission # Process: # 1. Streams data to other nodes # 2. Updates token ownership # 3. Leaves cluster # Time: 5 minutes to hours (depends on data size)

NEVER Skip Decommission!

Always decommission before removing a node!

  • ✅ Correct: decommission → wait → stop node
  • ❌ Wrong: Just stop/kill node
  • ⚠️ Skipping = permanent data loss!
🗑️

nodetool removenode

Force remove a dead node (emergency only!)

# First, get the host ID nodetool status # Remove dead node (run from live node) nodetool removenode # Check removal status nodetool removenode status # Force removal (if stuck) nodetool removenode force

⚠️ Use only when:

  • Node is permanently dead
  • Can't run decommission on it
  • Hardware failure, etc
🔄

nodetool rebuild

Stream data from another datacenter

# Rebuild from another DC nodetool rebuild -- datacenter1 # Rebuild specific keyspace nodetool rebuild -- datacenter1 -ks my_keyspace # Use when: # - Adding new datacenter # - Replacing failed node # - Restoring from another DC
📡

nodetool gossipinfo

View gossip state

# Show gossip information nodetool gossipinfo # Shows node states, heartbeat, etc # Useful for debugging cluster communication

💾 Backup Commands

Protect your data!

📸

nodetool snapshot

Create point-in-time backup

# Snapshot all keyspaces nodetool snapshot -t backup-20250106 # Specific keyspace nodetool snapshot -t backup-20250106 my_keyspace # Specific table nodetool snapshot -t backup-20250106 my_keyspace.users # List snapshots nodetool listsnapshots # Snapshots stored in: # /var/lib/cassandra/data/keyspace/table/snapshots/
🗑️

nodetool clearsnapshot

Delete old snapshots

# Clear all snapshots nodetool clearsnapshot # Clear specific snapshot nodetool clearsnapshot -t backup-20250106 # Clear snapshots for keyspace nodetool clearsnapshot my_keyspace # Automate cleanup (keep last 7 days) find /var/lib/cassandra/data -name snapshots \ -type d -mtime +7 -exec rm -rf {} \;

Backup Strategy

Recommended approach:

  1. Daily snapshots: Automated with cron
  2. Copy to S3/Object Storage: Off-server backup
  3. Test restores: Verify backups work
  4. Retention: Keep 7-30 days
  5. Monitor disk: Snapshots use space!

🔍 Troubleshooting Commands

Debug issues quickly!

🔍 Quick Health Check

# 1. Check all nodes are UP nodetool status # 2. Check heap usage nodetool info | grep "Heap Memory" # 3. Check pending compactions nodetool compactionstats # 4. Check thread pools nodetool tpstats # 5. Check schema agreement nodetool describecluster

🐌 Slow Queries

# Check latencies nodetool proxyhistograms # Check if compaction is lagging nodetool compactionstats # Check pending tasks nodetool tpstats # Enable query tracing in cqlsh # TRACING ON;

💾 High Disk Usage

# Check space per table nodetool tablestats -H # List snapshots (often the culprit) nodetool listsnapshots # Clear old snapshots nodetool clearsnapshot # Force major compaction (last resort) nodetool compact

🧠 High Memory Usage

# Check heap usage nodetool info # If > 75%: Need to increase heap or reduce load # Check cache sizes nodetool info | grep "Cache" # Trigger garbage collection (testing only) nodetool garbagecollect

🔌 Node Not Joining Cluster

# Check gossip state nodetool gossipinfo # Check if node sees others nodetool describecluster # Check cluster name matches grep "cluster_name" /etc/cassandra/cassandra.yaml # Check seeds are correct grep "seeds" /etc/cassandra/cassandra.yaml

💡 Best Practices

Use nodetool like a pro!

✅

DO

  • Run status daily
  • Schedule weekly repairs
  • Take daily snapshots
  • Monitor tpstats
  • Always decommission nodes
  • Use -H for human sizes
  • Check logs after commands
❌

DON'T

  • Run compact routinely
  • Skip decommissioning
  • Ignore high pending counts
  • Forget to clear snapshots
  • Run repairs too frequently
  • Force commands without checking
  • Ignore schema disagreements

Maintenance Schedule

Recommended routine:

  • Daily: Check nodetool status
  • Daily: Take snapshots (automated)
  • Weekly: Run nodetool repair
  • Weekly: Check tablestats, tpstats
  • Monthly: Clean up old snapshots
  • After scaling: Run cleanup
  • Before upgrades: Take snapshot, run flush

🎉 You're a Nodetool Expert!

Congratulations! You now know how to use nodetool effectively!

🎓 What You Learned:

  • ⭐ Essential commands: status, info, describecluster, ring
  • 🔧 Maintenance: repair, compact, cleanup, flush
  • 📊 Monitoring: tablestats, tpstats, proxyhistograms
  • 🌐 Cluster ops: decommission, removenode, rebuild
  • 💾 Backups: snapshot, clearsnapshot, listsnapshots
  • 🔍 Troubleshooting: Debug common issues
  • 💡 Best practices: Maintenance schedules, DO/DON'T

💡 Essential Commands Cheat Sheet:

# Daily checks nodetool status # Cluster health nodetool info # Node details # Weekly maintenance nodetool repair -full # Data consistency nodetool tablestats -H # Check stats # Monitoring nodetool tpstats # Thread pools nodetool proxyhistograms # Latencies # Backups nodetool snapshot -t backup-$(date +%Y%m%d) nodetool listsnapshots nodetool clearsnapshot # Emergency nodetool decommission # Remove node nodetool removenode # Force remove

🔧 Nodetool is your best friend for Cassandra operations!

Previous
Performance Monitoring
Advertisement

Responsive Ad