Skip to main content

Replacing All OSD Drives on a Rook Node

Problem​

All OSD drives on one node of a Rook-Ceph cluster must be replaced, for example to swap them for larger 7.6T drives. Every OSD on the node has to be removed, the disks wiped or swapped, and new OSDs created by the Rook operator. Then the same steps are repeated on the next node.

Doing this in the wrong order can cause an abrupt degraded state, a full cluster with all I/O blocked, or disks that Rook will not reuse because of stale LVM metadata.

All ceph commands below run in the Rook toolbox pod (kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash) or through the kubectl rook-ceph ceph ... plugin. See Management and Troubleshooting Tools.

To remove OSDs or a node permanently, without replacing the drives, see How to Remove OSDs from a Rook Ceph Cluster Safely.

Things to Check Before You Start​

Capacity​

Removing a whole node from a 4-node cluster removes 25% of the raw capacity until the new OSDs are in. The remaining 3 nodes must absorb that node's data without crossing the nearfull or full ratio. If an OSD reaches the full ratio, all writes to the affected pools stop.

ceph df
ceph osd df tree
ceph osd dump | grep ratio

As a rough check, the raw space used on the whole cluster must fit within the raw capacity of the remaining nodes, below the nearfull ratio (default 0.85), with headroom for uneven OSD fill. If it does not fit, stop and add capacity first.

Pool layout​

The data can only move to the remaining nodes if CRUSH can still place every replica or chunk there. With a host failure domain, check that:

  • Replicated pools have size ≤ the number of remaining hosts.
  • Erasure-coded pools have k+m ≤ the number of remaining hosts. An EC 2+2 pool on 4 hosts cannot rebuild onto 3 hosts, and its PGs stay undersized until the node returns.
ceph osd pool ls detail
ceph osd erasure-code-profile get <profile>

OSDs on the node​

ceph osd tree
ceph osd ls-tree <hostname>

Procedure​

Work on one node at a time. Do not start the next node until the cluster is HEALTH_OK and all PGs are active+clean.

Phase 1: Preparation and Evacuation​

  1. Cordon the node. Rook does not create OSDs on unschedulable nodes, so this stops the operator from recreating OSDs on the old disks while you work.

    kubectl cordon <node>

  2. Check that the OSDs can be stopped. Run this before scaling anything down. If the OSDs are already down, the check cannot tell you anything useful.

    ceph osd ok-to-stop osd.<id1> osd.<id2> ... osd.<idN>

    If it returns an error, find out which PGs would become inactive and fix that first.

  3. Mark the OSDs out. Ceph starts moving data to the other nodes while the current drives are still up and serving I/O, so redundancy is kept the whole time.

    ceph osd out osd.<id1> osd.<id2> ... osd.<idN>

    Ceph can recover if the OSDs simply go down, but then the cluster runs degraded until recovery is done. Marking them out first avoids that.

  4. Wait for the data to move. Watch until there are no backfilling, backfill_wait or remapped PGs and all PGs are active+clean.

    watch ceph -s

Phase 2: OSD Removal and Cleanup​

  1. Confirm that the OSDs no longer hold data that is needed:

    ceph osd safe-to-destroy osd.<id1> osd.<id2> ... osd.<idN>

    Do not continue until it reports that all OSDs are safe to destroy.

  2. Scale the Rook operator down, so it does not reconcile the OSD deployments while you remove them:

    kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0

  3. Scale down each OSD deployment:

    kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0

  4. Purge each OSD from the cluster. This removes it from the CRUSH map, deletes its auth key and removes it from the OSD map.

    ceph osd purge <id> --yes-i-really-mean-it

  5. Delete each OSD deployment:

    kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>

  6. Wipe the disks and clear the LVM metadata. Run this on the node itself. Rook creates OSDs with ceph-volume on LVM, and sgdisk and wipefs alone leave the device-mapper entries behind. Rook will then not treat the disk as clean.

    # Remove Ceph device-mapper (LVM) mappings
    dmsetup ls | grep ceph | awk '{print $1}' | xargs -r dmsetup remove
    rm -rf /dev/ceph-* /dev/mapper/ceph--*

    # Zap each disk
    sgdisk --zap-all /dev/sdX
    wipefs -a /dev/sdX
    dd if=/dev/zero of=/dev/sdX bs=1M count=100 oflag=direct,dsync
    partprobe /dev/sdX

    # Verify that no LVM or Ceph partitions remain
    lsblk

    The dmsetup command removes every Ceph mapping on the node. That is correct here only because every OSD on the node is being removed.

    If the old drives are physically removed and will not be reused, wiping is only needed to clear the device-mapper entries and to sanitize the drives. New raw drives need no wiping.

Phase 3: Hardware Replacement and Reintegration​

  1. Physically swap the drives for the new ones.

  2. If the CephCluster resource lists devices by name or path under storage.nodes, update it to match the new drives.

  3. Uncordon the node and start the operator again:

    kubectl uncordon <node>
    kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=1

  4. Rook runs the rook-ceph-osd-prepare-<node> job, finds the new raw drives, and creates the new OSDs. Check that they are up and in with the expected CRUSH weight (about 6.98 for a 7.6T drive):

    kubectl -n rook-ceph get pods -l app=rook-ceph-osd-prepare
    ceph osd tree

    If no new OSDs appear, check the logs of the prepare pod for that node.

  5. Wait for active+clean and HEALTH_OK before you move on to the next node.

Does Rook Handle This with Managed PDBs?​

When disruptionManagement.managePodBudgets: true is set in the CephCluster resource, Rook creates and manages PodDisruptionBudgets (PDBs) for the OSDs. They do not make this procedure safe on their own, and Rook does not set the flags this procedure needs.

What Rook does​

  • In normal operation Rook keeps a single rook-ceph-osd PDB that allows one OSD to be unavailable.
  • When OSDs in one failure domain (for example, one host) go down, Rook treats that failure domain as draining. It replaces the default PDB with per-failure-domain PDBs that allow disruptions only in the draining failure domain and block evictions everywhere else. This stops a second node from being drained at the same time.
  • Rook sets noout on the draining failure domain (ceph osd set-group noout <host>) for osdMaintenanceTimeout minutes (default 30). This is meant for short maintenance such as a reboot, so Ceph does not start moving data for a node that will come back.
  • When the OSDs are back and the PGs are active+clean (or pgHealthCheckTimeout runs out, if set), Rook restores the default PDB.

Why this does not cover a drive replacement​

  • PDBs only act on evictions, such as kubectl drain. kubectl cordon, kubectl scale --replicas=0 and kubectl delete deployment do not go through the eviction API, so no PDB stops them.
  • Rook never runs ceph osd ok-to-stop, safe-to-destroy, out or purge for you. Rook does not remove OSDs unless you enable removeOSDsIfOutAndSafeToRemove, and that only applies to OSDs that are already out.
  • The noout flag works against a permanent removal. It is right for a reboot but wrong when the data should move off the node. In this procedure the OSDs are marked out by hand before they go down, so noout has no effect on them. But if you scale OSDs down without marking them out first, noout keeps them in and the cluster stays degraded for up to 30 minutes before any recovery starts.
  • Rook does not set norebalance, nobackfill or similar flags, and does not turn the balancer off.

What to check​

After Phase 2, check that Rook did not leave a noout flag on the host and remove it if needed:

ceph health detail | grep -i noout
ceph osd unset-group noout <hostname>

Check the OSD PDBs. While the node is being worked on, it is expected to see the per-host PDBs instead of the default rook-ceph-osd PDB. After the new OSDs are active+clean, the default PDB should be back:

kubectl -n rook-ceph get pdb

If the per-host PDBs remain, other nodes cannot be drained. Check ceph -s for PGs that are not active+clean, and check the operator log.

References​