Configure Ceph for On-the-Wire Encryption

Problem

I would like to configure Ceph so that all network communications are encrypted.

Solution

First, ensure that all Ceph servers and clients are up to date with a supported release (Quincy or later).

By default, Ceph encrypts all communications with the MON and MGR, but does not encrypt communications on the data path (OSD, MDS).

In order to encrypt all communications between Ceph clients, mon, mgr, osd, and mds, use these configurations:

ceph config set global ms_client_mode secure
ceph config set global ms_cluster_mode secure
ceph config set global ms_service_mode secure
ceph config set global ms_mon_client_mode secure
ceph config set global ms_mon_cluster_mode secure
ceph config set global ms_mon_service_mode secure

Discussion

You can confirm that network traffic is secured by temporarily setting debug_ms to 1, and checking the logged network traffic for the secure flag. For example, from an OSD server host:

ceph config set osd.1 debug_ms 1
ceph config rm osd.1 debug_ms

Then check the log file of osd.1. Here is an example of a message which is not encrypted. Note the crc flag:

Sep 10 20:34:51 xxx ceph-osd[3021]: -- [xxx]:0/3345635598 <== osd.9 v2:[yyy]:6822/2305611088 166898 ==== osd_ping(ping_reply e233 up_from 100 ping_stamp 2024-09-10T20:34:51.016450+0000/534340.437500000s send_stamp 534340.062500000s delta_ub -0.376225382s) ==== 2033+0+0 (crc 0 0 0) 0x5654a85d5880 con 0x5654a66f4400

Here is an example of an encrypted message. Note the secure flag:

Sep 10 20:30:58 xxx ceph-osd[1654951]: -- [v2:[xxx]:6819/1860304126,v1:[xxx]:6821/1860304126] <== osd.7 [yyy]:0/470159112 18 ==== osd_ping(ping e233 up_from 99 ping_stamp 2024-09-10T20:30:58.601834+0000/534109.312500000s send_stamp 534109.312500000s delta_ub 534045.187500000s) ==== 2033+0+0 (secure 0 0 0) 0x564dd917c000 con 0x564dd96d0800