Skip to main content

Run CES with Rook on SUSE Virtualization (Harvester)

SUSE Virtualization (Harvester) runs on RKE2 Kubernetes, so CES can run on the Harvester nodes themselves with Rook. This page covers what is specific to Harvester. The installation itself follows Install CES with Rook.

There are two kinds of Ceph clients in a Harvester deployment, and they need different things:

  • Guest VMs mount CephFS or RBD themselves, with the kernel client of their own OS. Kubernetes and CSI are not involved.
  • Harvester itself uses Ceph for VM disks and volumes, through a ceph-csi driver running on the Harvester nodes.

Versions​

HarvesterRKE2 (Kubernetes)CES charts 1.0.0 (Rook 1.21.1)
v1.7v1.34supported
v1.8v1.35supported
v1.9v1.36supported

The Harvester version matters to Rook only through its Kubernetes version, which the charts accept from 1.32 to 1.37. Plan Harvester upgrades on their own merits, and read Harvester's release notes for known upgrade issues first.

Links on this page go to the Harvester v1.9 documentation, the latest release. Use the version selector on those pages to match your release.

Prepare the Harvester nodes​

The Harvester OS is immutable: most of the filesystem returns to its image on every reboot. Rook keeps the monitors' data and its configuration under /var/lib/rook, so that path must be persistent. Without it, a node reboot loses the monitor's store.

Add these settings to the Harvester configuration of every node, as Harvester documents for Rook:

os:
  persistent_state_paths:
    - /var/lib/rook
    - /var/lib/ceph
  modules:
    - rbd
    - nbd
  • New clusters: set them in the configuration at installation.
  • Existing clusters: Harvester does not re-read its installation configuration, and Update Harvester Configuration After Installation does not cover persistent paths or kernel modules. Ask SUSE for the supported way to add them to a running cluster before installing Rook.

Either way, reboot one node before installing Rook, and check that a test file in /var/lib/rook survives and that lsmod | grep rbd lists the module.

Choose the disks​

Harvester's own storage, SUSE Storage (Longhorn), also uses the nodes' disks. Give Rook only the disks meant for Ceph, by listing them in cluster.yaml:

rook-ceph-cluster:
  cephImage:
    tag: ces-v19.2.6-1
  cephClusterSpec:
    storage:
      useAllNodes: false
      useAllDevices: false
      nodes:
        - name: harvester-node-1
          devices:
            - name: /dev/disk/by-id/nvme-EXAMPLE_SERIAL_1

Node names are the Kubernetes node names (kubectl get nodes). Never leave useAllDevices: true on Harvester: Rook would take any empty disk, including one added for Longhorn but not yet in use.

Networking for guest VMs​

By default Rook's daemons listen on the Kubernetes pod network, which guest VMs cannot reach. If guest VMs are to mount Ceph directly, put the Ceph public network where they can reach it:

  • Host networking (cephClusterSpec.network.provider: host): the Ceph daemons listen on the Harvester nodes' own addresses. Guest VMs need a route to those addresses.
  • Multus (cephClusterSpec.network.provider: multus): the Ceph public network is a Multus network attachment, such as a VLAN network that the guest VMs are also attached to. Harvester already runs Multus. Follow Rook's network configuration for the requirements on the nodes.

The network provider cannot be changed after the cluster is created, so decide this before installing ces-ceph-cluster. To keep several CES clusters apart in one Harvester cluster, give each its own namespace and its own Ceph networks.

Guest VMs as clients​

Guest VMs use the Ceph kernel clients of their own OS:

  • RBD: the rbd module is part of the standard kernel on SLES and most Linux distributions.
  • CephFS on SLES 15 SP7: the CephFS module ships separately, as the cephfs-kmp-default package in the SLES 15 SP7 Legacy Module. Enable the module with SUSEConnect and install the package in the VM.
  • CephFS on other distributions: the module is usually part of the standard kernel.

Create a client key for the VMs that grants only what they need, for example for one CephFS filesystem:

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- \
  ceph fs authorize ceph-filesystem client.vm-app / rw

Then mount from the VM, with the monitors' addresses on the public network:

mount -t ceph <mon1>,<mon2>,<mon3>:/ /mnt/cephfs \
  -o name=vm-app,secretfile=/etc/ceph/vm-app.secret,fs=ceph-filesystem

Harvester VM disks on Ceph​

Harvester can store VM disks and volumes on Ceph through ceph-csi, as third-party storage (Harvester v1.5 and later).

  • RBD: Harvester validates the Rook RBD driver, rook-ceph.rbd.csi.ceph.com. Installing ces-csi-drivers in the rook-ceph namespace creates a driver of that name and the StorageClass ceph-block. Harvester supports it for VM images, boot and data disks, and live migration, but not for VM snapshots. Register the driver in Harvester's csi-driver-config setting, as the Harvester documentation describes.
  • CephFS: the Harvester OS kernel does not include the CephFS module, so CephFS volumes cannot be mounted by Harvester today. Clyso is working with SUSE on certifying the CephFS driver for Harvester v1.9, which depends on SUSE shipping that module. Contact Clyso before planning CephFS VM disks.

Limitations of third-party storage in Harvester apply to both: Harvester backs up VMs to a backup target only from Longhorn volumes. The third-party storage page for your Harvester version lists which features each validated driver supports.