Skip to main content

How to Remove OSDs from a Rook Ceph Cluster Safely

Problem​

One or more OSDs, or a whole OSD node, must be permanently removed from a Rook Ceph cluster.

Marking the OSDs out and deleting the Kubernetes worker is not enough on its own. That skips the checks that make sure no data is lost, and it leaves state behind in Ceph, in Rook and in Kubernetes:

  • The cluster can be degraded, or lose data, if the node goes away before backfill is done.
  • The node is still in the CephCluster spec, so the operator keeps trying to reconcile it.
  • The empty host bucket stays in the CRUSH map.
  • MON, MGR, MDS or RGW pods on the node only move after a timeout.
  • A noout flag set by Rook, OSD prepare jobs and /var/lib/rook remain.

This article describes the full procedure.

All ceph commands below run in the Rook toolbox pod:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Before You Start​

  1. Check the cluster health. Do not start while PGs are not active+clean:

    ceph status

  2. Find the OSDs to remove and the host they are on:

    ceph osd tree
    ceph osd ls-tree <hostname>

  3. Check capacity. The remaining OSDs must hold the removed OSDs' data and stay below the nearfull ratio (default 0.85). If an OSD reaches the full ratio, all writes to the affected pools stop.

    ceph df
    ceph osd df tree
    ceph osd dump | grep ratio

  4. If you are removing a whole node, check that CRUSH can still place all the data. With a host failure domain, every replicated pool's size and every erasure-coded pool's k+m must be no larger than the number of remaining hosts.

    ceph osd pool ls detail

Procedure​

The steps are the same for one OSD, several OSDs or a whole node. Steps marked Node removal only apply only when the node itself is going away.

1. Stop Rook from Recreating the OSDs​

Rook recreates an OSD on any disk it is allowed to use that still has OSD data on it. Before you remove anything, make sure Rook will not use the disks again:

  • If the CephCluster resource lists the devices or the node under storage.nodes, remove those entries.

  • If it uses useAllNodes: true or useAllDevices: true, cordon the node so that Rook does not create OSDs on it:

    kubectl cordon <node>

    Wipe the disks (step 7) before you uncordon the node.

2. Check That the OSDs Can Be Stopped​

Run this while the OSDs are still up:

ceph osd ok-to-stop osd.<id1> osd.<id2>

If it returns an error, fix that first.

3. Mark the OSDs Out and Wait for Backfill​

Marking the OSDs out while they are still running moves their data to other OSDs without the cluster ever becoming degraded.

ceph osd out osd.<id1> osd.<id2>

Wait until all PGs are active+clean:

watch ceph status

Then confirm that the OSDs no longer hold needed data:

ceph osd safe-to-destroy osd.<id1> osd.<id2>

Do not continue until all OSDs are reported as safe to destroy.

4. Stop the OSD Pods​

purge-osd only removes OSDs that are down. Scale each OSD deployment to 0:

kubectl -n rook-ceph scale deployment rook-ceph-osd-<id> --replicas=0

Check that the OSDs show as down:

ceph osd tree down

5. Purge the OSDs​

With the kubectl rook-ceph Krew plugin​

The kubectl rook-ceph plugin removes OSDs with one command. See Management and Troubleshooting Tools for how to install Krew. Then install the plugin:

kubectl krew install rook-ceph

Remove one OSD, or several OSDs as a comma-separated list:

kubectl rook-ceph rook purge-osd <id1>,<id2>

If Rook does not run in the rook-ceph namespace, add -n <cluster-ns> and --operator-namespace <operator-ns>.

The plugin runs rook ceph osd remove inside the operator pod, so the operator must be running. For each OSD it:

  1. Checks that the OSD is down. It refuses to remove an OSD that is up.
  2. Marks the OSD out.
  3. Runs ceph osd safe-to-destroy. If the OSD is not safe to destroy yet, it retries every minute until it is.
  4. Deletes the OSD deployment. For OSDs on PVCs, it also deletes the PVC and its prepare job.
  5. Runs ceph osd purge. This removes the OSD from the CRUSH map and the OSD map and deletes its auth key.
  6. Tries to remove the OSD's host from the CRUSH map. This only succeeds when the host has no OSDs left.

It does not change the CephCluster spec, wipe the disks, or remove the Kubernetes node.

warning

--force skips the safe-to-destroy check. Use it only for OSDs whose data is already lost, for example a failed disk, and only if ceph status shows no PGs that depend on them.

Without the plugin​

Run these for each OSD:

ceph osd purge <id> --yes-i-really-mean-it
kubectl -n rook-ceph delete deployment rook-ceph-osd-<id>

Rook also ships the same procedure as a Kubernetes job (deploy/examples/osd-purge.yaml in the Rook repository).

Check the result​

ceph osd tree
kubectl -n rook-ceph get deployment -l app=rook-ceph-osd

The OSDs must not appear in either list.

6. Remove the Node (Node Removal Only)​

  1. Move the other Ceph daemons off the node. List what still runs there:

    kubectl -n rook-ceph get pods -o wide --field-selector spec.nodeName=<node>

    Drain the node. Rook's OSD PodDisruptionBudgets allow one host at a time to be drained, so this works when no other host is being drained.

    kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

    If a MON ran on the node, wait for Rook to fail it over to another node and check that all MONs are in quorum:

    ceph mon stat

  2. Remove the now-empty host bucket from the CRUSH map. If you used purge-osd, it may already be gone. Check with ceph osd tree.

    ceph osd crush rm <hostname>

    The command fails if the host still has OSDs. That is a useful last check.

  3. Remove any noout flag that Rook set on the host while its OSDs were down:

    ceph health detail | grep -i noout
    ceph osd unset-group noout <hostname>

  4. Delete the node from Kubernetes:

    kubectl delete node <node>

7. Clean Up​

  1. Delete the leftover OSD prepare job:

    kubectl -n rook-ceph delete job rook-ceph-osd-prepare-<node>

  2. If the disks or the node will be used again, wipe the disks and remove the LVM mappings on the node. Also remove /var/lib/rook (or your dataDirHostPath) if the whole node is leaving the cluster. See Replacing All OSD Drives on a Rook Node for the wipe commands.

  3. Check that the default OSD PodDisruptionBudget is back:

    kubectl -n rook-ceph get pdb

    You should see rook-ceph-osd, not one PDB per host.

  4. Check the final state:

    ceph status
    ceph osd tree

Does Rook Do This Automatically?​

Mostly no. Rook's managed PodDisruptionBudgets only block pod evictions. They do not mark OSDs out, check safe-to-destroy or purge anything. When OSDs go down, Rook sets noout on their host, which delays recovery. For details, see Does Rook Handle This with Managed PDBs?

The CephCluster setting removeOSDsIfOutAndSafeToRemove: true makes Rook remove OSDs that are out and safe to destroy. It is off by default. Even when it is on, you still mark the OSDs out yourself, and you still clean up the node, CRUSH host and disks as described above.

References​