Secure Storage

Encryption at Rest

Protect stored data with transparent encryption!

Home › Security › Encryption at Rest

📖 The Story: Mark's Stolen Server Disaster

Mark ran Cassandra storing financial data. NO encryption at rest enabled. Server stolen from data center during a break-in. Thieves pulled out the drives. Took them home. Mounted drives on Linux box. Read EVERYTHING in plain text - account numbers, SSNs, transactions. Sold 500,000 records. $12M fine. Mark fired. Company sued. All because data on disk wasn't encrypted.

😱 The Physical Theft Attack

Friday Night - The Break-In:

  • 🔴 2:00 AM: Thieves break into data center
  • 🔴 They steal 1 Cassandra server (node3)
  • 🔴 Pull out 4 hard drives (2TB each)
  • 🔴 Walk out with 8TB of unencrypted data
  • 🔴 Alarm triggers but thieves already gone

Saturday Morning - Data Extraction:

-- What the thieves did (simple!): # 1. Connect stolen drive to Linux laptop $ sudo mount /dev/sdb1 /mnt/cassandra # 2. Navigate to data directory $ cd /mnt/cassandra/data/financial_db/accounts # 3. Use sstable2json tool (comes with Cassandra!) $ sstable2json ma-1-big-Data.db > accounts.json # 4. Open JSON file - SEE EVERYTHING IN PLAIN TEXT! $ cat accounts.json { "account_id": "123456", "name": "John Smith", "ssn": "123-45-6789", "balance": 50000, "credit_card": "4111-1111-1111-1111" ... } -- NO ENCRYPTION! All data readable! 💥 -- Extract 500,000 customer records in 2 hours!

Sunday - Records on Dark Web:

  • 💀 Thieves sell 500,000 records
  • 💀 $50 per record = $25M revenue for criminals
  • 💀 Includes: names, SSNs, accounts, balances, cards
  • 💀 Mark still doesn't know (thinks it's hardware failure)

Monday Morning - The Discovery:

  • 📞 Customer reports fraudulent charges
  • 📞 Then 100 more customers call
  • 📞 Then 1,000 more...
  • 🔍 Investigation: Server stolen, data extracted
  • 😱 Mark discovers: NO encryption at rest enabled!
  • 📰 Press: "Major Financial Data Breach"

The Damage:

  • 💰 $12M: Regulatory fine (no encryption)
  • ⚖️ $30M: Class action lawsuit
  • 📉 60%: Stock price drop
  • 😡 500,000: Victims of identity theft
  • 💼 Mark: Fired + banned from industry
  • 🏢 Company: Went out of business

✅ With Encryption at Rest

What Would Have Happened:

-- Same scenario, but WITH encryption at rest: # Thieves steal server, extract drives # Try to mount and read data... $ sudo mount /dev/sdb1 /mnt/cassandra $ cd /mnt/cassandra/data/financial_db/accounts $ sstable2json ma-1-big-Data.db > accounts.json $ cat accounts.json { "data": "a8f9e2c4d5b7f1a3e9d2..." ← ENCRYPTED! "data": "c7b4f9e1a8d3f2e6c9..." ← ENCRYPTED! "data": "9d3f1a2e8c7b5f4d1..." ← ENCRYPTED! } -- All data is ENCRYPTED! 🔐 -- Without encryption keys, it's useless! -- Thieves can't read anything! -- They try brute force: # AES-256 encryption = 2^256 possible keys # Would take billions of years to crack! 🎯 -- Attack FAILED! ✅

The Better Outcome:

  • ✅ $0: No breach, no fine
  • ✅ 0 records: Stolen (data encrypted)
  • ✅ Mark: Reports theft, replaces server
  • ✅ Customers: Data stays secure
  • ✅ Company: Business continues normally
  • ✅ Cost: $3K new server vs $42M loss

Encryption at rest: The difference between $42M loss and $3K! 🎯

💾 What Is Encryption at Rest?

Protect data on disk!

Definition

Encryption at rest means encrypting data stored on disk so that if someone gains physical access to the storage media, they cannot read the data without the encryption keys.

What gets encrypted:

  • SSTables: All data files
  • Commit logs: Write-ahead logs
  • Hints: Hinted handoff data
  • Saved caches: Row/key caches

What does NOT get encrypted:

  • System logs (can encrypt separately with OS tools)
  • Configuration files

Why You NEED Encryption at Rest

💿

Physical Theft

Threat: Stolen hardware

  • Server/laptop stolen
  • Drive theft during maintenance
  • Decommissioned drives
  • Data center break-in
  • Protection: Encryption makes data unreadable ✅
🗑️

Improper Disposal

Threat: Data recovery

  • Drives not properly wiped
  • Sold on eBay with data
  • Recycled without erasure
  • Backup tapes lost
  • Protection: Encrypted data is safe even if drive recovered ✅
⚖️

Compliance

Requirement: Regulations

  • HIPAA requires encryption
  • PCI-DSS requires encryption
  • GDPR recommends encryption
  • SOC 2 requires protection
  • Solution: Enable encryption at rest ✅
👤

Insider Threats

Threat: Rogue employees

  • IT admin copies drives
  • Contractor steals data
  • Disgruntled employee
  • Backup operator copies tapes
  • Protection: Data encrypted, keys separate ✅

⚙️ How Cassandra Encryption at Rest Works

Transparent Data Encryption (TDE)!

Architecture

Cassandra uses Transparent Data Encryption (TDE):

  • 📝 Write path: Data encrypted BEFORE writing to disk
  • 📖 Read path: Data decrypted AFTER reading from disk
  • 🔑 Keys: Master key → encrypts file-level keys
  • ⚡ Performance: Encryption happens in memory
  • ✨ Transparent: Applications don't see encryption

Encryption Layers

Layer 1: Master Key

The top-level key

  • Type: AES-256 symmetric key
  • Purpose: Encrypts all file-level keys
  • Storage: Java KeyStore OR external KMS (AWS KMS, HashiCorp Vault)
  • Rotation: Can rotate without re-encrypting data
  • Critical: If lost, ALL data is lost!

Layer 2: File-Level Keys (DEKs)

Data Encryption Keys

  • Type: AES-256 keys
  • Purpose: Encrypt individual SSTable files
  • Generation: Unique key per file
  • Storage: Encrypted with Master Key, stored in file header
  • Benefit: Each file independently encrypted

Layer 3: Encrypted Data

The actual encrypted bytes

  • Algorithm: AES-256-CBC or AES-256-GCM
  • Format: [Encrypted DEK Header][Encrypted Data]
  • On disk: All SSTables, commit logs, caches encrypted
  • In memory: Data decrypted when read

How Data Flows

/* WRITE PATH (with encryption): 1. App writes data to Cassandra 2. Data written to commit log (encrypted on disk) 3. Data written to memtable (in memory, NOT encrypted) 4. Memtable flushed to SSTable: a. Generate unique DEK for this SSTable b. Encrypt DEK with Master Key c. Write encrypted DEK to file header d. Encrypt data with DEK e. Write encrypted data to disk 5. SSTable on disk is FULLY ENCRYPTED */ /* READ PATH (with encryption): 1. App queries Cassandra 2. Cassandra reads SSTable from disk 3. Read encrypted DEK from file header 4. Decrypt DEK using Master Key 5. Decrypt data using DEK 6. Return decrypted data to app All transparent to application! ✅ */

🔧 Setting Up Encryption at Rest

Complete configuration guide!

IMPORTANT: Cassandra 4.0+ Required

Native transparent encryption at rest is available in Cassandra 4.0 and later.

  • ⚠️ Cassandra 3.x and earlier: No native support
  • ✅ Cassandra 4.0+: Built-in TDE support
  • 🔄 Alternative for 3.x: Use filesystem-level encryption (LUKS, dm-crypt)
1

Generate Master Key

Create encryption key in Java KeyStore

-- Generate a master encryption key: $ keytool -genseckey \ -alias cassandra_encryption_key \ -keyalg AES \ -keysize 256 \ -keystore /etc/cassandra/conf/.encryption_keystore \ -storetype JCEKS \ -storepass "your_strong_password" -- Secure the keystore file: $ sudo chown cassandra:cassandra /etc/cassandra/conf/.encryption_keystore $ sudo chmod 600 /etc/cassandra/conf/.encryption_keystore -- CRITICAL: Backup this keystore! $ sudo cp /etc/cassandra/conf/.encryption_keystore \ /secure/backup/location/.encryption_keystore.backup -- If you lose this key, ALL DATA IS LOST FOREVER! 💥
2

Configure cassandra.yaml

Enable encryption in config

-- Edit /etc/cassandra/cassandra.yaml on ALL nodes: # Transparent Data Encryption transparent_data_encryption_options: enabled: true chunk_length_kb: 64 ← Encryption block size cipher: AES/CBC/PKCS5Padding ← Or AES/GCM/NoPadding key_provider: JKS ← Java KeyStore # KeyStore configuration keystore: /etc/cassandra/conf/.encryption_keystore keystore_password: your_strong_password store_type: JCEKS key_alias: cassandra_encryption_key -- Alternative: Use AWS KMS (if in AWS) transparent_data_encryption_options: enabled: true key_provider: AWS_KMS aws_kms_region: us-east-1 aws_kms_key_id: arn:aws:kms:us-east-1:123456789012:key/abc-def-... -- Save and exit
3

Enable for Specific Tables (Optional)

Can enable per-table or cluster-wide

-- Option 1: Enable cluster-wide (in cassandra.yaml) # Sets default for all new tables -- Option 2: Enable per table (via CQL) ALTER TABLE my_keyspace.sensitive_data WITH compression = { 'enabled': true, 'class': 'org.apache.cassandra.io.compress.LZ4Compressor' } AND encryption = { 'enabled': true }; -- Check encryption status: SELECT keyspace_name, table_name, encryption FROM system_schema.tables WHERE keyspace_name = 'my_keyspace';
4

Rolling Restart Cluster

Apply configuration

-- Restart nodes ONE AT A TIME: # Node 1: $ nodetool drain $ sudo systemctl restart cassandra $ nodetool status ← Wait for UN # Wait 2-5 minutes, then node 2... # Continue for all nodes -- After restart, NEW data is encrypted -- OLD data still unencrypted (need to rewrite)
5

Re-encrypt Existing Data

Encrypt data that was already on disk

-- Option 1: Run compaction (forces rewrite) $ nodetool compact my_keyspace sensitive_table -- This reads old SSTables and writes new encrypted ones -- Takes time for large tables! -- Option 2: Upgrade SSTables (if upgrading Cassandra) $ nodetool upgradesstables my_keyspace -- Option 3: Run scrub (rewrites all SSTables) $ nodetool scrub my_keyspace sensitive_table -- Monitor progress: $ nodetool compactionstats -- After completion, verify old SSTables removed

🔑 Key Management Best Practices

Critical for security!

1. Key Storage Options

Java KeyStore (JKS) - Basic:

  • ✅ Simple to set up
  • ✅ No external dependencies
  • ⚠️ Key stored on same server as data
  • ⚠️ Need to backup carefully
  • 📁 Good for: Small deployments, dev/test

External KMS - Production:

  • ✅ Keys separate from data
  • ✅ Automatic rotation
  • ✅ Audit logging
  • ✅ Access control
  • 🌐 Options: AWS KMS, Azure Key Vault, HashiCorp Vault
  • 📁 Good for: Production, compliance requirements

2. Key Backup & Recovery

-- CRITICAL: Backup encryption keys! # Backup keystore to secure location: $ sudo cp /etc/cassandra/conf/.encryption_keystore \ /secure/backup/.encryption_keystore.$(date +%Y%m%d) # Encrypt the backup itself: $ gpg --symmetric --cipher-algo AES256 \ /secure/backup/.encryption_keystore.20240115 # Store in multiple locations: # - Off-site backup server # - Cloud storage (S3, encrypted) # - Physical safe -- Test recovery periodically: 1. Restore keystore from backup 2. Try to start Cassandra with it 3. Verify data readable

3. Key Rotation

-- Rotate master key annually (best practice): # 1. Generate new key in keystore $ keytool -genseckey \ -alias cassandra_encryption_key_2025 \ -keyalg AES -keysize 256 \ -keystore /etc/cassandra/conf/.encryption_keystore # 2. Update cassandra.yaml key_alias: cassandra_encryption_key_2025 # 3. Rolling restart cluster # 4. Re-encrypt existing data (optional but recommended) $ nodetool compact my_keyspace # 5. Old key still needed to read old SSTables # Don't delete until all data re-encrypted!

4. Key Access Control

  • File permissions: chmod 600 on keystore
  • Ownership: Only cassandra user can read
  • Password: Strong password, not in config
  • Audit: Log all key access
  • Separation: Keys on different mount than data
  • Least privilege: Only DBAs have key access

KEY LOSS = DATA LOSS

If you lose the master encryption key, ALL encrypted data is PERMANENTLY LOST!

  • 💥 No recovery: Impossible to decrypt without key
  • 💥 No backdoor: AES-256 is unbreakable
  • 💥 Entire cluster: All data unusable
  • ✅ Solution: Multiple secure backups in different locations
  • ✅ Test: Verify backup recovery regularly

📊 Performance Impact

What to expect!

Metric Impact Notes
CPU Usage +10-20% Encryption/decryption overhead
Read Latency +5-10% Decrypt on read
Write Latency +5-10% Encrypt on write
Throughput -10-15% Less ops/sec
Disk Space +2-5% Encryption overhead
Memory Minimal Encryption in-memory, minimal impact

Mitigation Strategies

🔧

Hardware Acceleration

  • AES-NI: Intel/AMD CPU instruction set
  • Performance: 3-10x faster encryption
  • Enable: Most modern CPUs have it
  • Check: grep aes /proc/cpuinfo
  • Result: Minimal performance impact
💻

Right-Size Resources

  • Add 10-20% more CPU capacity
  • Ensure adequate memory
  • Use SSDs for better I/O
  • Monitor and adjust
  • Cost: Small increase for big security

Real-World Impact

With modern hardware (AES-NI), encryption impact is minimal:

  • ✅ Most users see < 5% performance degradation
  • ✅ Security benefit FAR outweighs small performance cost
  • ✅ $42M breach cost >> $5K/month extra hardware
  • ✅ Compliance requirements make it mandatory anyway

✅ Testing & Verification

Confirm encryption is working!

1. Check Cassandra Logs

-- Look for encryption initialization: $ grep -i encryption /var/log/cassandra/system.log INFO Transparent data encryption enabled INFO Using key provider: JKS INFO Encryption cipher: AES/CBC/PKCS5Padding -- Should see "encryption enabled" message ✅

2. Verify SSTable Encryption

-- Try to read SSTable file directly: $ cd /var/lib/cassandra/data/my_keyspace/sensitive_table-*/ $ ls -lh -rw-r--r-- ma-1-big-Data.db (50 MB) -- Without encryption: readable text $ strings ma-1-big-Data.db | head /* Would see: John Smith 123-45-6789 ...actual data... */ -- With encryption: gibberish $ strings ma-1-big-Data.db | head /* Should see: a8f9e2c4d5b7... c7b4f9e1a8d3... ...encrypted binary... → Cannot read actual data! ✅ */

3. Test Data Accessibility

-- Query via CQL (should work normally): $ cqlsh cqlsh> SELECT * FROM my_keyspace.sensitive_table LIMIT 1; /* Should return data normally: account_id | name | ssn ------------+------------+------------- 123456 | John Smith | 123-45-6789 → Data accessible via Cassandra ✅ → But NOT readable from disk ✅ */

4. Test Key Dependency

-- Simulate key loss (TESTING ONLY!): # 1. Stop Cassandra $ sudo systemctl stop cassandra # 2. Rename keystore (simulate loss) $ sudo mv /etc/cassandra/conf/.encryption_keystore \ /etc/cassandra/conf/.encryption_keystore.bak # 3. Try to start Cassandra $ sudo systemctl start cassandra /* Should FAIL with: ERROR: Cannot load encryption key Failed to start cassandra.service → Proves data is actually encrypted! ✅ */ # 4. Restore keystore $ sudo mv /etc/cassandra/conf/.encryption_keystore.bak \ /etc/cassandra/conf/.encryption_keystore $ sudo systemctl start cassandra -- Should start successfully and data accessible again

💡 Encryption at Rest Best Practices

Do it right!

✅

DO

  • Enable encryption in production
  • Use AES-256 encryption
  • Backup encryption keys (multiple locations)
  • Use external KMS in production
  • Test key recovery regularly
  • Rotate keys annually
  • Enable hardware AES acceleration
  • Monitor performance impact
❌

DON'T

  • Run production without encryption
  • Store keys with data
  • Use weak passwords
  • Forget to backup keys
  • Lose encryption keys
  • Skip testing
  • Ignore performance monitoring
  • Use same key forever

Complete Encryption Checklist

Task Details Done?
Generate Master Key AES-256 key in keystore ☐
Backup Keys Multiple secure locations ☐
Configure Encryption cassandra.yaml updated ☐
Enable on Cluster Rolling restart applied ☐
Re-encrypt Existing Compact all tables ☐
Test Encryption Verify SSTables encrypted ☐
Test Recovery Restore from backup works ☐
Secure Keystore chmod 600, proper ownership ☐
Document Process Key location, recovery procedure ☐
Compliance Meets HIPAA/PCI/GDPR requirements ☐

Common Issues & Solutions

Issue Cause Solution
Node won't start Can't find keystore Check path in cassandra.yaml
Key not found error Wrong alias Verify key_alias matches keystore
Performance degradation No AES-NI acceleration Enable in BIOS, check CPU support
Data still readable Old SSTables not re-encrypted Run nodetool compact
Key rotation failed Old key still needed Keep old key until data re-encrypted

🎉 Master Encryption at Rest!

You now know how to protect stored Cassandra data with encryption!

🎓 What You Learned:

  • 📖 Mark's disaster: $42M loss from stolen unencrypted server
  • 💾 What it is: Transparent Data Encryption (TDE) for data on disk
  • ⚙️ How it works: Master key → file keys → encrypted data
  • 🔧 Setup: Complete 5-step configuration process
  • 🔑 Key management: Backup, rotation, access control
  • 📊 Performance: 10-20% impact, mitigated with AES-NI
  • ✅ Testing: 4 methods to verify encryption working
  • 💡 Best practices: External KMS, regular backups, testing

💡 Key Takeaways:

  1. Enable in production - Physical security is not enough
  2. Backup encryption keys - Multiple secure locations
  3. Use external KMS - Separate keys from data
  4. Test recovery - Verify backup keys work
  5. Monitor performance - Enable AES-NI acceleration
  6. Compliance requirement - HIPAA/PCI/GDPR need it

📋 Quick Setup (5 Steps):

# 1. Generate master key keytool -genseckey -alias cassandra_key ... # 2. Configure cassandra.yaml transparent_data_encryption_options: enabled: true keystore: /path/to/.encryption_keystore # 3. Backup keys (CRITICAL!) cp .encryption_keystore /secure/backup/ # 4. Rolling restart nodetool drain && systemctl restart cassandra # 5. Re-encrypt existing data nodetool compact my_keyspace # Done! Data encrypted! 💾🔐

⚠️ CRITICAL REMINDER:

💥 KEY LOSS = TOTAL DATA LOSS! 💥
Backup encryption keys to multiple secure locations!
Test recovery regularly!

💾 Remember Mark: Encryption at rest = $42M saved! 🎯
Enable encryption NOW!

Previous
Role Management
Advertisement

Responsive Ad