Skip to main content

Enabling RocksDB Compression on Ceph OSDs

Overview​

RocksDB compression can significantly reduce the database size for Ceph OSDs using BlueStore, reducing the risk of RocksDB data spilling over to the slow device and degrading OSD performance.. This guide demonstrates how to enable compression on a specific OSD and verify the compression effectiveness through monitoring and compaction.

RocksDB Compression can be enabled on Ceph starting from the Pacific release.

Prerequisites​

  • Access to the Ceph cluster with admin privileges
  • An OSD with RocksDB compression currently disabled
  • Monitoring tools available (watch, grep)

Procedure​

Step 1: Select Target OSD​

Identify an OSD that currently has RocksDB compression disabled. Replace X with your actual OSD ID in all subsequent commands.

Step 2: Monitor Database Size​

Before making changes, monitor the current database size to establish a baseline. Run the following command to check the database metrics every 2 seconds:

watch "ceph tell osd.X perf dump bluefs | grep -E 'db_total_bytes|db_used_bytes'"

Example output:

"db_total_bytes": 2575294464
"db_used_bytes": 106954752

Important: Record the db_used_bytes value. This will be your baseline for comparing compression effectiveness.

Step 3: Enable RocksDB Compression​

Enable RocksDB compression for the specific OSD using the LZ4 compression algorithm:

ceph config set osd.X bluestore_rocksdb_options_annex compression=kLZ4Compression

This command sets the compression option for the specified OSD. LZ4 compression provides a good balance between compression ratio and performance.

Step 4: Restart the OSD​

Restart the OSD to apply the compression configuration:

ceph orch daemon restart osd.X

Wait for the OSD to complete its restart and return to an up state before proceeding.

Step 5: Trigger Compaction​

Trigger manual compaction twice while monitoring the BlueFS performance metrics. Compaction is necessary to apply compression to existing data:

ceph tell osd.X compact

Wait for the first compaction to complete, then run the command again:

ceph tell osd.X compact

Step 6: Observe Compression Results​

While running the compaction commands, continue monitoring the database size using the watch command from Step 2.

Expected behavior:

  1. During the compaction process, db_used_bytes will initially increase
  2. After compaction completes, db_used_bytes will drop significantly below the initial baseline value recorded in Step 2

The reduction in db_used_bytes indicates successful compression of the RocksDB database.

Verification​

To verify compression is enabled and active:

# Check the current configuration
ceph config get osd.X bluestore_rocksdb_options_annex

# Compare current db_used_bytes to your baseline
ceph tell osd.X perf dump bluefs | grep -E 'db_total_bytes|db_used_bytes'

Notes​

  • Compression is applied to new data immediately after the configuration change and OSD restart
  • Existing data requires manual compaction to be compressed
  • Running compaction twice ensures thorough compression of the database
  • The temporary increase in db_used_bytes during compaction is normal and expected
  • LZ4 compression (kLZ4Compression) offers fast compression/decompression with reasonable compression ratios

Additional Compression Algorithms​

Besides LZ4, RocksDB supports other compression algorithms:

  • kNoCompression - Disable compression (default)
  • kSnappyCompression - Snappy compression
  • kZlibCompression - Zlib compression (higher compression ratio, slower)
  • kLZ4Compression - LZ4 compression (recommended)
  • kLZ4HCCompression - LZ4 High Compression
  • kZSTD - Zstandard compression

Choose the compression algorithm based on your performance requirements and storage constraints.