Install Clyso Enterprise Storage (CES) with Rook
Clyso publishes Helm charts that install CES on Kubernetes through Rook. Each chart wraps an upstream Rook or ceph-csi chart at an exact version, so one release of the charts is one tested combination of Rook, ceph-csi and the CES image. The charts keep upstream's defaults apart from the CES image and a few safety settings, so Rook's documentation applies to every setting.
| Chart | Installs | Wraps |
|---|---|---|
ces-rook-operator | the Rook operator and ceph-csi-operator, once per Kubernetes cluster | rook-ceph 1.21.1 |
ces-csi-drivers | the Ceph CSI drivers, only if pods in this cluster use Ceph volumes | ceph-csi-drivers 1.1.0 |
ces-ceph-cluster | a CES cluster, one release per cluster | rook-ceph-cluster 1.21.1 |
The charts are in the Helm repository https://charts.ces.clyso.com, and
their source, with a README for each chart, is at
github.com/clyso/ces-helm-charts.
Earlier versions of this page installed the latest upstream Rook operator
with a values file from github.com/clyso/ces-manifests. That combination is
not pinned and can mix Rook versions. Use the charts below instead. To move
an existing installation onto the charts, see
Moving existing upstream releases.
Prerequisites
- A running Kubernetes cluster (v1.25+ recommended).
- Validation (Recommended): Qualify your Kubernetes environment using the Clyso Kubernetes Analyzer to ensure storage and network readiness.
- Nodes with raw storage (unformatted disks) for OSDs.
kubectlandhelminstalled.- The CES community edition image must be available in a registry accessible by
your Kubernetes
nodes (e.g.,
http://harbor.clyso.com/ces/ceph/ceph:ces-v25.03.2).
The CES images are public: harbor.clyso.com/ces/ceph/ceph. Air-gapped sites
mirror that image, and the CSI images that each chart's README lists.
Step 1: Describe the cluster
Write cluster.yaml. It names the CES release the cluster runs, which the
chart requires, and the nodes and disks Rook may use:
rook-ceph-cluster:
cephImage:
tag: ces-v19.2.6-1
cephClusterSpec:
storage:
useAllNodes: false
useAllDevices: false
nodes:
- name: node-1
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c301
- name: node-2
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c302
- name: node-3
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c303
Upstream's default is useAllNodes: true and useAllDevices: true: Rook
then turns every empty disk on every node into an OSD. List the nodes and
devices explicitly, by /dev/disk/by-id/ path, and set both to false.
Upstream's defaults also create a replicated block pool with the default
StorageClass ceph-block, a CephFS filesystem ceph-filesystem and an
object store ceph-objectstore, each spread over 3 hosts. The
ces-ceph-cluster README
lists the defaults to review.
Step 2: Install the charts
Install the three charts in this order, at the same version:
helm repo add clyso https://charts.ces.clyso.com
helm repo update clyso
helm install ces-rook-operator clyso/ces-rook-operator --version 1.0.0 \
-n rook-ceph --create-namespace --wait
helm install ces-csi-drivers clyso/ces-csi-drivers --version 1.0.0 \
-n rook-ceph --wait
helm install ces-ceph-cluster clyso/ces-ceph-cluster --version 1.0.0 \
-n rook-ceph -f cluster.yaml
Skip ces-csi-drivers if no pods in this Kubernetes cluster will use Ceph
volumes, for instance when only clients outside Kubernetes use the cluster.
Step 3: Verify the installation
Wait for the cluster to report HEALTH_OK:
kubectl -n rook-ceph get cephcluster -w
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
Check that every Ceph daemon runs the CES image:
kubectl -n rook-ceph get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
| tr ' ' '\n' | sort -u
Upgrades
Upgrade the three charts together, operator first, and wait for
HEALTH_OK after each step. A chart upgrade never changes the CES release
a cluster runs; that is a change of cephImage.tag in cluster.yaml, made
on its own. Never uninstall ces-rook-operator or ces-csi-drivers while a
CephCluster exists: that deletes the CSI resources of every cluster. See
Upgrading.
Troubleshooting
- Helm refuses the Kubernetes version: the charts accept only the range their Rook release is tested on. Upgrade Kubernetes, or ask Clyso for the charts matching your version.
- Operator in CrashLoopBackOff: check its logs with
kubectl -n rook-ceph logs deploy/rook-ceph-operator. "failed the ceph version check" means the image's Ceph release is not one this Rook supports: Rook 1.21 runs CES releases based on Squid (19.2) and Tentacle (20.2.1 or later). - No OSDs created: Rook skips disks with partitions or a filesystem. Check
the
rook-ceph-osd-preparepod logs for the node.