Skip to main content

Install Clyso Enterprise Storage (CES) with Rook

Clyso publishes Helm charts that install CES on Kubernetes through Rook. Each chart wraps an upstream Rook or ceph-csi chart at an exact version, so one release of the charts is one tested combination of Rook, ceph-csi and the CES image. The charts keep upstream's defaults apart from the CES image and a few safety settings, so Rook's documentation applies to every setting.

ChartInstallsWraps
ces-rook-operatorthe Rook operator and ceph-csi-operator, once per Kubernetes clusterrook-ceph 1.21.1
ces-csi-driversthe Ceph CSI drivers, only if pods in this cluster use Ceph volumesceph-csi-drivers 1.1.0
ces-ceph-clustera CES cluster, one release per clusterrook-ceph-cluster 1.21.1

The charts are in the Helm repository https://charts.ces.clyso.com, and their source, with a README for each chart, is at github.com/clyso/ces-helm-charts.

note

Earlier versions of this page installed the latest upstream Rook operator with a values file from github.com/clyso/ces-manifests. That combination is not pinned and can mix Rook versions. Use the charts below instead. To move an existing installation onto the charts, see Moving existing upstream releases.

Prerequisites​

  • A running Kubernetes cluster (v1.25+ recommended).
  • Validation (Recommended): Qualify your Kubernetes environment using the Clyso Kubernetes Analyzer to ensure storage and network readiness.
  • Nodes with raw storage (unformatted disks) for OSDs.
  • kubectl and helm installed.
  • The CES community edition image must be available in a registry accessible by your Kubernetes nodes (e.g., http://harbor.clyso.com/ces/ceph/ceph:ces-v25.03.2).

The CES images are public: harbor.clyso.com/ces/ceph/ceph. Air-gapped sites mirror that image, and the CSI images that each chart's README lists.

Step 1: Describe the cluster​

Write cluster.yaml. It names the CES release the cluster runs, which the chart requires, and the nodes and disks Rook may use:

rook-ceph-cluster:
cephImage:
tag: ces-v19.2.6-1
cephClusterSpec:
storage:
useAllNodes: false
useAllDevices: false
nodes:
- name: node-1
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c301
- name: node-2
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c302
- name: node-3
devices:
- name: /dev/disk/by-id/wwn-0x5000c500a1b2c303
warning

Upstream's default is useAllNodes: true and useAllDevices: true: Rook then turns every empty disk on every node into an OSD. List the nodes and devices explicitly, by /dev/disk/by-id/ path, and set both to false.

Upstream's defaults also create a replicated block pool with the default StorageClass ceph-block, a CephFS filesystem ceph-filesystem and an object store ceph-objectstore, each spread over 3 hosts. The ces-ceph-cluster README lists the defaults to review.

Step 2: Install the charts​

Install the three charts in this order, at the same version:

helm repo add clyso https://charts.ces.clyso.com
helm repo update clyso

helm install ces-rook-operator clyso/ces-rook-operator --version 1.0.0 \
-n rook-ceph --create-namespace --wait
helm install ces-csi-drivers clyso/ces-csi-drivers --version 1.0.0 \
-n rook-ceph --wait
helm install ces-ceph-cluster clyso/ces-ceph-cluster --version 1.0.0 \
-n rook-ceph -f cluster.yaml

Skip ces-csi-drivers if no pods in this Kubernetes cluster will use Ceph volumes, for instance when only clients outside Kubernetes use the cluster.

Step 3: Verify the installation​

Wait for the cluster to report HEALTH_OK:

kubectl -n rook-ceph get cephcluster -w
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

Check that every Ceph daemon runs the CES image:

kubectl -n rook-ceph get pods -o jsonpath='{.items[*].spec.containers[*].image}' \
| tr ' ' '\n' | sort -u

Upgrades​

Upgrade the three charts together, operator first, and wait for HEALTH_OK after each step. A chart upgrade never changes the CES release a cluster runs; that is a change of cephImage.tag in cluster.yaml, made on its own. Never uninstall ces-rook-operator or ces-csi-drivers while a CephCluster exists: that deletes the CSI resources of every cluster. See Upgrading.

Troubleshooting​

  • Helm refuses the Kubernetes version: the charts accept only the range their Rook release is tested on. Upgrade Kubernetes, or ask Clyso for the charts matching your version.
  • Operator in CrashLoopBackOff: check its logs with kubectl -n rook-ceph logs deploy/rook-ceph-operator. "failed the ceph version check" means the image's Ceph release is not one this Rook supports: Rook 1.21 runs CES releases based on Squid (19.2) and Tentacle (20.2.1 or later).
  • No OSDs created: Rook skips disks with partitions or a filesystem. Check the rook-ceph-osd-prepare pod logs for the node.