Rook
CES and Ceph KB articles related to Rook, covering Rook orchestration and troubleshooting.
Rook Ceph Support Matrix
rook version | supported ceph version name | supported ceph versions | source |
---|---|---|---|
v1.17 | Ceph Squid | v19.2.0 or newer | rook.io v1.17 |
v1.17 | Ceph Reef | v18.2.0 or newer | rook.io v1.17 |
v1.16 | Ceph Squid | v19.2.0 or newer | rook.io v1.16 |
v1.16 | Ceph Reef | v18.2.0 or newer | rook.io v1.16 |
v1.15 | Ceph Squid | v19.2.0 or newer | rook.io v1.15 |
v1.15 | Ceph Reef | v18.2.0 or newer | rook.io v1.15 |
v1.15 | Ceph Quincy | v17.2.0 or newer | rook.io v1.15 |
v1.14 | Ceph Reef | v18.2.0 or newer | rook.io v1.14 |
v1.14 | Ceph Quincy | v17.2.0 or newer | rook.io v1.14 |
v1.13 | Ceph Reef | v18.2.0 or newer | rook.io v1.13 |
v1.13 | Ceph Quincy | v17.2.0 or newer | rook.io v1.13 |
v1.12 | Ceph Reef | v18.2.0 or newer | rook.io v1.12 |
v1.12 | Ceph Quincy | v17.2.0 or newer | rook.io v1.12 |
v1.12 | Ceph Pacific | v16.2.7 or newer | rook.io v1.12 |
v1.11 | Ceph Quincy | v17.2.0 or newer | rook.io v1.11 |
v1.11 | Ceph Pacific | v16.2.0 or newer | rook.io v1.11 |
v1.10 | Ceph Quincy | v17.2.0 or newer | rook.io v1.10 |
v1.10 | Ceph Pacific | v16.2.0 or newer | rook.io v1.10 |
v1.9 | Ceph Quincy | v18.2.0 or newer | rook.io v1.9 |
v1.9 | Ceph Pacific | v17.2.0 or newer | rook.io v1.9 |
v1.9 | Ceph Octopus | v16.2.7 or newer | rook.io v1.9 |
The Ceph operator was declared stable in December 2018 in the Rook v0.9 release, providing a production storage platform for many years. Rook is hosted by the Cloud Native Computing Foundation (CNCF) as a graduated level project.
Management and troubleshooting tools
For troubleshooting and managing a rook ceph cluster users can make use of Rook-Ceph Krew plugin. This tool lets you run common Ceph commands directly using kubectl, without needing to connect to the toolbox pod.
Installing the kubectl
Krew Plugin
To use the kubectl rook-ceph
plugin, you must first install Krew.
Follow the instructions below for macOS/Linux, or refer to the official guide for the most recent instructions.
Krew Installation (macOS/Linux - Bash or Zsh)
Prerequisite: Ensure that git
is installed.
Run the following command to download and install Krew:
(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' \
-e 's/\(arm\)\(64\)\?.*/\1\2/' \
-e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
Then, add Krew to your PATH
by appending this line to your ~/.bashrc
or ~/.zshrc
:
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
Restart your shell and verify the installation:
kubectl krew
Install the rook-ceph
Plugin
Use Krew to install the Rook Ceph plugin:
kubectl krew install rook-ceph
Using kubectl rook-ceph
You can now use the plugin to interact with your Ceph cluster without the toolbox pod.
Check Ceph Health
kubectl rook-ceph ceph health detail
List OSDs and MONs
kubectl rook-ceph ceph osd tree
kubectl rook-ceph ceph mon dump
Monitor Rook and Ceph CRs During Upgrades
kubectl rook-ceph ceph status
kubectl rook-ceph status a
For more usage examples, see the official repository: https://github.com/rook/kubectl-rook-ceph
Top 5 useful Rook commands
toolbox execution
kubectl -n rook-ceph exec -it deployments/rook-ceph-tools -- /bin/bash
rook pod list
kubectl -n rook-ceph get pod -o wide
search in osd pod logs
search pattern: omap
kubectl -n rook-ceph get pods | grep "osd-" | egrep -v "prepare" |awk '{print $1}'` ; do kubectl -n rook-ceph logs $i | grep omap; done
use ceph daemon command inside an osd pod
exec into the osd.9 pod and show the help of ceph daemon
kubectl exec -it -n rook-ceph <osd-pod-name> -- bash
unset CEPH_ARGS; ceph daemon /var/run/ceph/ceph-osd.9.asok help
restart rook operator
kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0
kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=1
How to Use ceph-bluestore-tool
and ceph-objectstore-tool
in Rook Ceph — and Where to Find Them
In Rook Ceph, advanced tools like ceph-bluestore-tool
and ceph-objectstore-tool
are essential for inspecting, debugging, or repairing an OSD’s internal data. These tools are not available in the standard OSD pods due to their limited container environments. Instead, Rook provides a special maintenance mode to safely access and use these tools.
Step 1: Start OSD Maintenance Mode
Use the rook-ceph
Krew plugin to place an OSD into maintenance mode. This temporarily stops the OSD and starts a dedicated pod with the same data mounted and extra tools enabled.
kubectl rook-ceph -n rook-ceph maintenance start rook-ceph-osd-0
This does the following:
- Scales down the original
rook-ceph-osd-0
Deployment - Waits for the current pod to fully terminate
- Starts a new pod with the label
rook-ceph-osd-0-maintenance
, containing the necessary tools
Step 2: Verify the Maintenance Pod is Running
Run:
kubectl get pods -n rook-ceph
You should see a pod similar to:
rook-ceph-osd-0-maintenance-66cc78cd87-cxqgt 2/2 Running 0 35s
This pod has:
- The Ceph container with full access to
ceph-bluestore-tool
,ceph-objectstore-tool
Similarly for monstore tool mon deployment can be started in maintenance mode.
Step 3: Access the Tools
Execute a shell into the Ceph container:
kubectl exec -it -n rook-ceph rook-ceph-osd-0-maintenance-66cc78cd87-cxqgt -- ceph-objectstore-tool -h
From inside the container, you can now run:
ceph-bluestore-tool -h
After the maintainenance is done, make sure to bring the OSD back up using :
kubectl rook-ceph maintenance stop rook-ceph-osd-0
Note: These are powerful tools primarily for debugging, repair, and manual inspection.
Why Use Maintenance Mode
Using Rook’s maintenance mode:
- Allows safe manipulation of on-disk data
- Provides access to Ceph internal tools not available during regular pod operation
- Is ideal for low-level debugging and recovery workflows in production environments