Skip to main content

Understanding CRC and Secure Messenger Mode Options

Problem​

Ceph's network protocols support two communication modes:

  • crc: network traffic is checksummed for integrity
  • secure: network traffic is encrypted

These are controlled by six options, but is unclear what they do.

Solution​

Ceph network communications are rather complex, allowing to select different modes for different channels of communication, e.g. connections with the MON and MGR daemons and connections with OSD and MDS daemons. The options also allow to differentiate modes for intra-cluster and client connections.

In detail, Ceph has six messenger mode configuration options.

The first two options are read by Ceph clients, such as RBD and CephFS clients and the ceph command line tools. Note that the RGW is also a ceph client.

  • When a client connects to a MON or MGR, it reads its ms_mon_client_mode option for its preferred order of allowed connection modes. The default is "secure crc", meaning that a client prefers to encrypt connections with the MON and MGR, but will fall back to crc-only if the MON/MGR does not allow secure mode.
  • When a client is connecting to any other daemon (e.g. an OSD or MDS), it reads the ms_client_mode option to decide that prefered order. The default is "crc secure", meaning that the client prefers those connections to be in crc mode, but will fall back to secure mode if the OSD/MDS does not allow crc mode.

The next options are read by the MON and MGR daemons.

  • When a MON/MGR is connecting to another MON/MGR, it reads its ms_mon_cluster_mode option. The default is "secure crc", meaning that all intra-MON/MGR traffic is encrypted by default, but that any MON/MGR allowing only crc mode will also be permitted.
  • When a MON/MGR is connecting with anything else (e.g. a client, OSD, or MDS), then it reads its ms_mon_service_mode option to determine the list of allowed modes. The default is "secure crc", meaning that MON and MGR daemons allow peers to connect in either secure or crc mode.

Lastly, the other Ceph daemons (OSD and MDS) proceed as follows:

  • When an OSD/MDS is connecting with a MON or MGR, it reads its ms_mon_cluster_mode option, which defaults to "secure crc". This means that an OSD/MDS prefers secure mode when communicating with MONs and MGRs, but will fall back to crc if the MON/MGR does not allow secure mode.
  • When an OSD/MDS is communicating with other OSD or MDS daemons, it reads the ms_cluster_mode option. This defaults to "secure crc", which means that the that daemon prefers secure connections, but will fall back to crc if the other side does not allow secure mode.
  • When an OSD/MDS is communicating with a client, it reads its ms_service_mode option to determine which modes it allows for such connections. This defaults to "crc secure", meaning that OSD and MDS daemons allow both crc and secure mode connections with clients.