Skip to main content

How to use ceph cephadm set-extra-ceph-conf correctly

Problem​

I need to set a custom ceph.conf for all the daemons in my cephadm managed cluster. How do I do it?

Solution​

  1. Write your extra ceph.conf option in a file my.conf, for example:
[osd]
ms_async_op_threads=1

  1. Use ceph cephadm set-extra-ceph-conf -i my.conf to set the extra ceph.conf options.

  2. Inspect the cephadm log with ceph log last cephadm: You should see that the orchestrator is reconfiguring all daemons in the cluster with the new ceph.conf options. For example:

2024-08-01T23:56:24.506532+0000 mgr.cph-test-josh-1.rbsnzr (mgr.14514) 103 : cephadm [INF] Reconfiguring osd.9 (extra config changed)...
2024-08-01T23:56:24.510089+0000 mgr.cph-test-josh-1.rbsnzr (mgr.14514) 104 : cephadm [INF] Reconfiguring daemon osd.9 on cph-test-josh-3
...

  1. Verify that the config has changed, looking on any host in /var/lib/ceph/<fsid>/<daemon>/config.

  2. Now restart daemons as needed to apply the updated ceph.conf file.

Discussion​

Ceph options stored via ceph config set ... are stored in the ceph-mon database. Some Ceph options, e.g. several ms_* options, need to be set in ceph.conf before connecting to any mon. Those options can be set using this set-extra-ceph-conf feature.