Skip to main content

Rook

CES and Ceph KB articles related to Rook, covering Rook orchestration and troubleshooting.

Rook Ceph Support Matrix​

rook versionsupported ceph version namesupported ceph versionssource
v1.17Ceph Squidv19.2.0 or newerrook.io v1.17
v1.17Ceph Reefv18.2.0 or newerrook.io v1.17
v1.16Ceph Squidv19.2.0 or newerrook.io v1.16
v1.16Ceph Reefv18.2.0 or newerrook.io v1.16
v1.15Ceph Squidv19.2.0 or newerrook.io v1.15
v1.15Ceph Reefv18.2.0 or newerrook.io v1.15
v1.15Ceph Quincyv17.2.0 or newerrook.io v1.15
v1.14Ceph Reef v18.2.0 or newerrook.io v1.14
v1.14Ceph Quincyv17.2.0 or newerrook.io v1.14
v1.13Ceph Reef v18.2.0 or newerrook.io v1.13
v1.13Ceph Quincyv17.2.0 or newerrook.io v1.13
v1.12Ceph Reef v18.2.0 or newerrook.io v1.12
v1.12Ceph Quincyv17.2.0 or newerrook.io v1.12
v1.12Ceph Pacific v16.2.7 or newerrook.io v1.12
v1.11Ceph Quincyv17.2.0 or newerrook.io v1.11
v1.11Ceph Pacific v16.2.0 or newerrook.io v1.11
v1.10Ceph Quincyv17.2.0 or newerrook.io v1.10
v1.10Ceph Pacific v16.2.0 or newerrook.io v1.10
v1.9Ceph Quincy v18.2.0 or newerrook.io v1.9
v1.9Ceph Pacificv17.2.0 or newerrook.io v1.9
v1.9Ceph Octopus v16.2.7 or newerrook.io v1.9

The Ceph operator was declared stable in December 2018 in the Rook v0.9 release, providing a production storage platform for many years. Rook is hosted by the Cloud Native Computing Foundation (CNCF) as a graduated level project.

Management and troubleshooting tools​

For troubleshooting and managing a rook ceph cluster users can make use of Rook-Ceph Krew plugin. This tool lets you run common Ceph commands directly using kubectl, without needing to connect to the toolbox pod.

Installing the kubectl Krew Plugin​

To use the kubectl rook-ceph plugin, you must first install Krew. Follow the instructions below for macOS/Linux, or refer to the official guide for the most recent instructions.

Krew Installation (macOS/Linux - Bash or Zsh)​

Prerequisite: Ensure that git is installed.

Run the following command to download and install Krew:

(
  set -x; cd "$(mktemp -d)" &&
  OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
  ARCH="$(uname -m | sed -e 's/x86_64/amd64/' \
                         -e 's/\(arm\)\(64\)\?.*/\1\2/' \
                         -e 's/aarch64$/arm64/')" &&
  KREW="krew-${OS}_${ARCH}" &&
  curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
  tar zxvf "${KREW}.tar.gz" &&
  ./"${KREW}" install krew
)

Then, add Krew to your PATH by appending this line to your ~/.bashrc or ~/.zshrc:

export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"

Restart your shell and verify the installation:

kubectl krew

Install the rook-ceph Plugin​

Use Krew to install the Rook Ceph plugin:

kubectl krew install rook-ceph

Using kubectl rook-ceph​

You can now use the plugin to interact with your Ceph cluster without the toolbox pod.

Check Ceph Health​

kubectl rook-ceph ceph health detail

List OSDs and MONs​

kubectl rook-ceph ceph osd tree
kubectl rook-ceph ceph mon dump

Monitor Rook and Ceph CRs During Upgrades​

kubectl rook-ceph ceph status
kubectl rook-ceph status a

For more usage examples, see the official repository: https://github.com/rook/kubectl-rook-ceph

Top 5 useful Rook commands​

toolbox execution​

kubectl -n rook-ceph exec -it deployments/rook-ceph-tools -- /bin/bash

rook pod list​

kubectl -n rook-ceph get pod  -o wide

search in osd pod logs​

search pattern: omap

kubectl -n rook-ceph get pods | grep "osd-" | egrep -v "prepare" |awk '{print $1}'` ; do kubectl -n rook-ceph logs $i | grep omap; done

use ceph daemon command inside an osd pod​

exec into the osd.9 pod and show the help of ceph daemon

kubectl exec -it -n rook-ceph <osd-pod-name> -- bash
unset CEPH_ARGS; ceph daemon /var/run/ceph/ceph-osd.9.asok help

restart rook operator​

kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0
kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=1

How to Use ceph-bluestore-tool and ceph-objectstore-tool in Rook Ceph — and Where to Find Them​

In Rook Ceph, advanced tools like ceph-bluestore-tool and ceph-objectstore-tool are essential for inspecting, debugging, or repairing an OSD’s internal data. These tools are not available in the standard OSD pods due to their limited container environments. Instead, Rook provides a special maintenance mode to safely access and use these tools.

Step 1: Start OSD Maintenance Mode​

Use the rook-ceph Krew plugin to place an OSD into maintenance mode. This temporarily stops the OSD and starts a dedicated pod with the same data mounted and extra tools enabled.

kubectl rook-ceph -n rook-ceph maintenance start rook-ceph-osd-0

This does the following:

  • Scales down the original rook-ceph-osd-0 Deployment
  • Waits for the current pod to fully terminate
  • Starts a new pod with the label rook-ceph-osd-0-maintenance, containing the necessary tools

Step 2: Verify the Maintenance Pod is Running​

Run:

kubectl get pods -n rook-ceph

You should see a pod similar to:

rook-ceph-osd-0-maintenance-66cc78cd87-cxqgt   2/2   Running   0   35s

This pod has:

  • The Ceph container with full access to ceph-bluestore-tool, ceph-objectstore-tool

Similarly for monstore tool mon deployment can be started in maintenance mode.

Step 3: Access the Tools​

Execute a shell into the Ceph container:

kubectl exec -it -n rook-ceph rook-ceph-osd-0-maintenance-66cc78cd87-cxqgt -- ceph-objectstore-tool -h

From inside the container, you can now run:

ceph-bluestore-tool -h

After the maintainenance is done, make sure to bring the OSD back up using :

kubectl rook-ceph maintenance stop rook-ceph-osd-0

Note: These are powerful tools primarily for debugging, repair, and manual inspection.

Why Use Maintenance Mode​

Using Rook’s maintenance mode:

  • Allows safe manipulation of on-disk data
  • Provides access to Ceph internal tools not available during regular pod operation
  • Is ideal for low-level debugging and recovery workflows in production environments