Skip to main content

BlueStore Compression Ratio Limitation and Workaround

Overview​

BlueStore compression is currently limited to a maximum 2:1 (2x) compression ratio when using default settings. This limitation is caused by the default bluestore_compression_min_blob_size_hdd configuration value. This article explains why this limitation exists, how to work around it, and what improvements are coming in future Ceph releases.

Why the 2x Compression Ratio Limit Exists​

The consistent 2:1 compression ratio limitation is caused by the default bluestore_compression_min_blob_size_hdd setting. With the default configuration, BlueStore uses 8K blobs for compression. Due to this small blob size, the compression system cannot exceed a 2x compression ratio, regardless of how compressible the data actually is.

This limitation affects all data stored on HDDs using BlueStore with default compression settings, even when the data could theoretically achieve much higher compression ratios with algorithms like LZ4 or Zstandard.

Current Workaround​

Until the permanent fix is merged into Ceph, you can achieve better compression ratios by increasing the bluestore_compression_min_blob_size_hdd value and restarting the affected OSDs.

Step 1: Increase the Minimum Blob Size​

Set the minimum blob size to a larger value, such as 64K:

ceph config set osd bluestore_compression_min_blob_size_hdd 65536

Note: 65536 bytes = 64 KB

Alternatively, you can set this for specific OSDs:

ceph config set osd.X bluestore_compression_min_blob_size_hdd 65536

Step 2: Restart the OSDs​

For the configuration change to take effect, restart the affected OSDs:

# Restart all OSDs (if setting was applied globally)
ceph orch restart osd

# Or restart a specific OSD
ceph orch daemon restart osd.X

Step 3: Verify Configuration​

Confirm the new setting is active:

# Check global setting
ceph config get osd bluestore_compression_min_blob_size_hdd

# Check specific OSD setting
ceph config get osd.X bluestore_compression_min_blob_size_hdd

Step 4: Monitor Compression Ratio​

After restarting the OSDs, new data written to the cluster will use the larger blob size and can achieve better compression ratios. Monitor the compression effectiveness:

ceph osd df

Look for the compression ratio in the output to verify improved compression on newly written data.

Important Considerations​

Existing Data​

Changing bluestore_compression_min_blob_size_hdd only affects new data written after the OSDs are restarted. Existing compressed data will retain its original compression characteristics until it is rewritten.

Performance Impact​

Larger blob sizes may have performance implications:

  • Write Performance: Larger blobs may require more memory and CPU for compression operations
  • Read Performance: Reading compressed data may be more efficient with larger blobs due to fewer decompression operations
  • Memory Usage: Larger blobs increase memory requirements for compression buffers

Test the performance impact in your environment before applying this change to production clusters.

Choosing the Right Value​

The optimal bluestore_compression_min_blob_size_hdd value depends on your workload:

  • 64 KB (65536): Good balance for most workloads, recommended starting point
  • 128 KB (131072): May provide better compression for large sequential writes
  • 32 KB (32768): More conservative increase if 64 KB causes issues

Future Improvements​

Two significant improvements are coming to BlueStore compression in future Ceph releases:

Permanent Fix for Compression Ratio Limit​

A permanent fix that removes the 2x compression ratio limitation is currently in review. This fix will allow BlueStore to achieve higher compression ratios without requiring manual configuration changes. Once merged, this will improve compression efficiency for all users automatically.

Related PR: 67421

Dynamic Configuration Without Restarts​

In the near future, changing BlueStore compression settings will no longer require OSD restarts. This improvement will allow administrators to adjust compression parameters dynamically, making it easier to tune compression settings without service disruption.

Related PR: 67433

These improvements will make BlueStore compression more effective and easier to manage in production environments.

Verification​

To verify that your compression ratio has improved after applying the workaround:

Check Compression Statistics​

# View compression statistics per OSD
ceph daemon osd.X perf dump bluestore

# Look for compression-related metrics
ceph daemon osd.X perf dump | grep compress

Monitor Cluster-Wide Compression​

# View overall compression effectiveness
ceph osd pool stats

Compare the compression ratio before and after the change to confirm improvement.

Summary​

  • Current Limitation: Default BlueStore configuration limits compression to 2x ratio due to 8K blob size
  • Workaround: Increase bluestore_compression_min_blob_size_hdd to 64K (65536 bytes) and restart OSDs
  • Future: Permanent fixes coming that will remove the limitation and eliminate the need for OSD restarts
  • Impact: Only affects new data; existing data retains original compression characteristics

By applying this workaround, you can achieve significantly better compression ratios on new data while awaiting the permanent fixes in upcoming Ceph releases.