Skip to main content

General

CES and Ceph KB articles related to general Ceph topics, such as configuration or the messenger, etc.

How can I configure Ceph for on-the-wire encryption?​

Problem​

I would like to configure Ceph so that all network communications are encrypted.

Solution​

First, ensure that all Ceph servers and clients are up to date with a supported release (Quincy or later).

By default, Ceph encrypts all communications with the MON and MGR, but does not encrypt communications on the data path (OSD, MDS).

In order to encrypt all communications between Ceph clients, mon, mgr, osd, and mds, use these configurations:

ceph config set global ms_client_mode secure
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_mon_client_mode secure
ceph config set global ms_mon_cluster_mode secure
ceph config set global ms_mon_service_mode secure

Discussion​

You can confirm that network traffic is secured by temporarily setting debug_ms to 1, and checking the logged network traffic for the secure flag. For example, from an OSD server host:

ceph config set osd.1 debug_ms 1
ceph config rm osd.1 debug_ms

Then check the log file of osd.1. Here is an example of a message which is not encrypted. Note the crc flag:

Sep 10 20:34:51 xxx ceph-osd[3021]: -- [xxx]:0/3345635598 <== osd.9 v2:[yyy]:6822/2305611088 166898 ==== osd_ping(ping_reply e233 up_from 100 ping_stamp 2024-09-10T20:34:51.016450+0000/534340.437500000s send_stamp 534340.062500000s delta_ub -0.376225382s) ==== 2033+0+0 (crc 0 0 0) 0x5654a85d5880 con 0x5654a66f4400

Here is an example of an encrypted message. Note the secure flag:

Sep 10 20:30:58 xxx ceph-osd[1654951]: -- [v2:[xxx]:6819/1860304126,v1:[xxx]:6821/1860304126] <== osd.7 [yyy]:0/470159112 18 ==== osd_ping(ping e233 up_from 99 ping_stamp 2024-09-10T20:30:58.601834+0000/534109.312500000s send_stamp 534109.312500000s delta_ub 534045.187500000s) ==== 2033+0+0 (secure 0 0 0) 0x564dd917c000 con 0x564dd96d0800

What is the meaning of the crc and secure messenger mode options?​

Problem​

Ceph's network protocols support two communication modes:

  • crc: network traffic is checksummed for integrity
  • secure: network traffic is encrypted

These are controlled by six options, but is unclear what they do.

Solution​

Ceph network communications are rather complex, allowing to select different modes for different channels of communication, e.g. connections with the MON and MGR daemons and connections with OSD and MDS daemons. The options also allow to differentiate modes for intra-cluster and client connections.

In detail, Ceph has six messenger mode configuration options.

The first two options are read by Ceph clients, such as RBD and CephFS clients and the ceph command line tools. Note that the RGW is also a ceph client.

  • When a client connects to a MON or MGR, it reads its ms_mon_client_mode option for its preferred order of allowed connection modes. The default is "secure crc", meaning that a client prefers to encrypt connections with the MON and MGR, but will fall back to crc-only if the MON/MGR does not allow secure mode.
  • When a client is connecting to any other daemon (e.g. an OSD or MDS), it reads the ms_client_mode option to decide that prefered order. The default is "crc secure", meaning that the client prefers those connections to be in crc mode, but will fall back to secure mode if the OSD/MDS does not allow crc mode.

The next options are read by the MON and MGR daemons.

  • When a MON/MGR is connecting to another MON/MGR, it reads its ms_mon_cluster_mode option. The default is "secure crc", meaning that all intra-MON/MGR traffic is encrypted by default, but that any MON/MGR allowing only crc mode will also be permitted.
  • When a MON/MGR is connecting with anything else (e.g. a client, OSD, or MDS), then it reads its ms_mon_service_mode option to determine the list of allowed modes. The default is "secure crc", meaning that MON and MGR daemons allow peers to connect in either secure or crc mode.

Lastly, the other Ceph daemons (OSD and MDS) proceed as follows:

  • When an OSD/MDS is connecting with a MON or MGR, it reads its ms_mon_cluster_mode option, which defaults to "secure crc". This means that an OSD/MDS prefers secure mode when communicating with MONs and MGRs, but will fall back to crc if the MON/MGR does not allow secure mode.
  • When an OSD/MDS is communicating with other OSD or MDS daemons, it reads the ms_cluster_mode option. This defaults to "secure crc", which means that the that daemon prefers secure connections, but will fall back to crc if the other side does not allow secure mode.
  • When an OSD/MDS is communicating with a client, it reads its ms_service_mode option to determine which modes it allows for such connections. This defaults to "crc secure", meaning that OSD and MDS daemons allow both crc and secure mode connections with clients.