Skip to main content

Cephadm

CES and Ceph KB articles related to Cephadm, covering Cephadm orchestration and troubleshooting.

Deploy Ceph using Cephadm with IPv6​

Problem​

Cephadm failed when we tried to configure the cluster with IPv6,

 ./cephadm bootstrap --mon-ip xxxx:xxxx:xx:xxxx:x::xx --skip-monitoring-stack -c ceph.conf
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chrony.service is enabled and running
Repeating the final host check...
podman (/usr/bin/podman) version 4.7.2 is present
systemctl is present
lvcreate is present
Unit chrony.service is enabled and running
Host looks OK
Cluster fsid: 19c15c1a-c125-11ee-a33d-17acafae1852
...
The public CIDR network xxxx:xxx:xx:xxxx:x::/80 (from -c conf file) is not configured locally.
ERROR: None of the public CIDR network(s) {'xxxx:xxx:xx:xxxx:x::/80'} (from -c conf file) is configured locally.

Solution​

Check the ip of the node and bootstrap the cluster skipping mon network, as we will configure it after bootstrap.

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet6 xxxx**:xxx:xx:xxxx:x::xx/128** scope global
       valid_lft forever preferred_lft forever
    inet6 xxx::x/xx scope link
       valid_lft forever preferred_lft forever
$ ./cephadm bootstrap --mon-ip xxxx:xxx:xx:xxxx:x::xx --skip-mon-network
  • Configure mon network after a successful bootstrap from cephadm shell, the network should be in CIDR notation.
$ ./cephadm shell -- ceph config set mon public_network xxxx:xxx:xx:xxxx:x::/80
  • Add the nodes as usual and follow upstream Ceph documentation for deploying other daemons, if the daemons are not able to join the cluster, please check are the ports open or not.

How to use ceph cephadm set-extra-ceph-conf correctly​

Problem​

I need to set a custom ceph.conf for all the daemons in my cephadm managed cluster. How do I do it?

Solution​

  1. Write your extra ceph.conf option in a file my.conf, for example:
[osd]
ms_async_op_threads=1

  1. Use ceph cephadm set-extra-ceph-conf -i my.conf to set the extra ceph.conf options.

  2. Inspect the cephadm log with ceph log last cephadm: You should see that the orchestrator is reconfiguring all daemons in the cluster with the new ceph.conf options. For example:

2024-08-01T23:56:24.506532+0000 mgr.cph-test-josh-1.rbsnzr (mgr.14514) 103 : cephadm [INF] Reconfiguring osd.9 (extra config changed)...
2024-08-01T23:56:24.510089+0000 mgr.cph-test-josh-1.rbsnzr (mgr.14514) 104 : cephadm [INF] Reconfiguring daemon osd.9 on cph-test-josh-3
...

  1. Verify that the config has changed, looking on any host in /var/lib/ceph/<fsid>/<daemon>/config.

  2. Now restart daemons as needed to apply the updated ceph.conf file.

Discussion​

Ceph options stored via ceph config set ... are stored in the ceph-mon database. Some Ceph options, e.g. several ms_* options, need to be set in ceph.conf before connecting to any mon. Those options can be set using this set-extra-ceph-conf feature.

Setting extra-ceph-conf just hangs on the CLI​

Problem​

When I use ceph cephadm set-extra-ceph-conf or ceph cephadm get-extra-ceph-conf the command line just hangs forever.

Solution​

Running ceph cephadm set-extra-ceph-conf without any option can corrupt the local value stored and cause the commands to hang. This can be cleaned by removing the internal ceph config-key setting and then restarting the active ceph-mgr:

$ ceph config-key ls
[
    mgr/cephadm/extra_ceph_conf,
]
$ ceph config-key get mgr/cephadm/extra_ceph_conf
{"conf": null, "last_modified": "2024-08-01T23:50:20.959362Z"}
$ ceph config-key rm mgr/cephadm/extra_ceph_conf
key deleted
$ ceph mgr fail