Skip to main content

Configure Ceph for On-the-Wire Encryption

Problem​

I would like to configure Ceph so that all network communications are encrypted.

Solution​

First, ensure that all Ceph servers and clients are up to date with a supported release (Quincy or later).

By default, Ceph encrypts all communications with the MON and MGR, but does not encrypt communications on the data path (OSD, MDS).

In order to encrypt all communications between Ceph clients, mon, mgr, osd, and mds, use these configurations:

ceph config set global ms_client_mode secure
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_mon_client_mode secure
ceph config set global ms_mon_cluster_mode secure
ceph config set global ms_mon_service_mode secure

Discussion​

Confirming that network traffic is secured​

You can confirm that network traffic is secured by temporarily setting debug_ms to 1, and checking the logged network traffic for the secure flag.

For example, run the following commands from an OSD server host:

ceph config set osd.1 debug_ms 1
ceph config rm osd.1 debug_ms

Then check the log file of osd.1. Here is an example of a message which is not encrypted. Note the crc flag:

Sep 10 20:34:51 xxx ceph-osd[3021]: -- [xxx]:0/3345635598 <== osd.9 v2:[yyy]:6822/2305611088 166898 ==== osd_ping(ping_reply e233 up_from 100 ping_stamp 2024-09-10T20:34:51.016450+0000/534340.437500000s send_stamp 534340.062500000s delta_ub -0.376225382s) ==== 2033+0+0 (crc 0 0 0) 0x5654a85d5880 con 0x5654a66f4400

Here is an example of an encrypted message. Note the secure flag:

Sep 10 20:30:58 xxx ceph-osd[1654951]: -- [v2:[xxx]:6819/1860304126,v1:[xxx]:6821/1860304126] <== osd.7 [yyy]:0/470159112 18 ==== osd_ping(ping e233 up_from 99 ping_stamp 2024-09-10T20:30:58.601834+0000/534109.312500000s send_stamp 534109.312500000s delta_ub 534045.187500000s) ==== 2033+0+0 (secure 0 0 0) 0x564dd917c000 con 0x564dd96d0800

CRC mode and Secure mode - differences​

Messenger v2 supports two modes for communication: crc and secure.

  • crc confirms only that the data is correctly sent without errors
  • secure confirms that the data is correctly sent without errors and also encrypts the network traffic

There are three different types of client/server component in a Ceph cluster:

  1. The CLIENTS: for example: Ceph cli, RADOS cli, librados, librbd, ceph-fuse, kernel Ceph (CephFS or RBD), RADOS cli, Ceph cli, RGW daemons.
  2. The MON and MGR daemons (ceph-mon, ceph-mgr)
  3. The OSD and MDS daemons (ceph-osd, ceph-mds)

There are six different configuration options used to control when crc or secure mode is used to facilitate communicate between the clients, mon and mgr daemons, and osd and mds daemons:

  • ms_client_mode
  • ms_cluster_mode
  • ms_service_mode
  • ms_mon_client_mode
  • ms_mon_cluster_mode
  • ms_mon_service_mode

The default Ceph configurations for these options result in the following situation:

  1. Communication between the MON/MGR and everything else is secure (encrypted).

    • CLIENTS < - > MON/MGR is secure
    • MON/MGR < - > MON/MGR is secure
    • MON/MGR < - > OSD/MDS is secure

  2. All other communications are in crc mode:

    • CLIENT < - > OSD/MDS
    • OSD/MDS < - > OSD/MDS This means that by default your data is not encrypted while in transport.

Data is encrypted using AES-128-GCM and HMAC-SHA256.

If you want all traffic to be encrypted (meaning that you want all connections to be in secure mode and you want crc mode to be rejected), run the following commands:

ceph config set global ms_mon_cluster_mode secure
ceph config set global ms_mon_service_mode secure
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure

Further remarks​

As far as the Ceph Messenger is concerned, RGW is a client.

The S3 protocol is not a Ceph native protocol.

S3 is based on HTTP/HTTPS.

Enabling SSL on all RGW daemons might negatively impact the performance of client I/O. Requiring a network daemon or client to encrypt all traffic increases the CPU workload on both sides during encryption and decryption. The capability of your hardware is what determines whether I/O performance will slow. See https://ceph.io/en/news/blog/2023/ceph-encryption-performance/ for more on this. In this linked example, the impact on the tested cluster was not crippling.

See also this presentation about Cephx, which contains information about changes made in 2025 to Cephx mode: 2025 Cephalocon presentation about Cephx