Skip to main content

Tuning Ceph OSDs on ARM Servers

Problem​

I am deploying an all-NVMe Ceph cluster on ARM (aarch64) servers. With the default configuration, OSD performance is lower than expected compared to similar x86 hardware. Which Ceph version and configuration options should I use?

Solution​

The configuration below is a starting point for ARM server processors (e.g. Ampere Altra, NVIDIA Grace, Huawei Kunpeng 920, AWS Graviton) with many cores, lower per-core performance than current x86 CPUs, and several NUMA nodes per server. Confirm the values with benchmarks on your own hardware before rolling them out to production (see FIO Commands for Storage Performance Tests and Adapting to Other ARM Platforms).

The recommended version for ARM deployments is Ceph Tentacle v20.2.2 (or later v20.2.x). Before upgrading, review the Tentacle known bugs page.

Host Tuning​

On every OSD host, enable the network-latency tuned profile:

# dnf install tuned        # or: apt install tuned
# systemctl enable --now tuned
# tuned-adm profile network-latency
# tuned-adm active
Current active profile: network-latency

The profile persists across reboots. See the Discussion below for what it changes.

OptionRecommendedDefaultSet viaRestart needed
ms_async_op_threads53set-extra-ceph-confYes
osd_numa_auto_affinitytruetrueceph config setYes
osd_op_queuewpqmclock_schedulerceph config setYes
osd_memory_target20G4Gceph config setNo
osd_op_num_threads_per_shard_ssd42ceph config setYes
mon_target_pg_per_osd250100ceph config setNo
  1. Set ms_async_op_threads in the [osd] section of ceph.conf. This option is read when the messenger starts, before the OSD connects to the MONs, so it must be set with ceph cephadm set-extra-ceph-conf rather than ceph config set. Write the option to a file arm.conf:
[osd]
ms_async_op_threads = 5

and apply it:

# ceph cephadm set-extra-ceph-conf -i arm.conf

See Using ceph cephadm set-extra-ceph-conf for how to verify that cephadm has reconfigured the daemons.

  1. Set the remaining OSD options in the config database:
# ceph config set osd osd_numa_auto_affinity true
# ceph config set osd osd_op_queue wpq
# ceph config set osd osd_memory_target 20G
# ceph config set osd osd_op_num_threads_per_shard_ssd 4
  1. Raise the PG autoscaler target:
# ceph config set global mon_target_pg_per_osd 250

Check that mon_max_pg_per_osd leaves enough headroom above the new target (see the Discussion below):

# ceph config get mon mon_max_pg_per_osd
  1. Restart the OSDs, one failure domain at a time, waiting for HEALTH_OK between each:
# ceph orch daemon restart osd.<id>

or, for a whole host:

# for osd in $(ceph osd ls-tree <host>); do ceph orch daemon restart osd.$osd; done

Verification​

Confirm the running values on an OSD:

# ceph config show osd.0 | egrep 'ms_async_op_threads|osd_numa_auto_affinity|osd_op_queue |osd_memory_target |osd_op_num_threads_per_shard_ssd'

Confirm the tuned profile on every OSD host:

# tuned-adm active
Current active profile: network-latency
# tuned-adm verify

Confirm NUMA pinning for every OSD:

# ceph osd numa-status
OSD  HOST    NETWORK  STORAGE  AFFINITY  CPUS
  0  node01        0        0         0  0-47
  1  node01        0        0         0  0-47
  2  node01        1        1         1  48-95
...

The AFFINITY column must be populated for every OSD. An empty AFFINITY column means the OSD is running unpinned.

The OSD also logs its NUMA decision at startup. Check the log of each OSD, e.g.:

# cephadm logs --name osd.0 | grep set_numa_affinity
... set_numa_affinity storage numa node 0
... set_numa_affinity public network ens1f0np0 numa node 0
... set_numa_affinity cluster network ens1f1np1 numa node 0
... set_numa_affinity setting numa affinity to node 0 cpus 0-47

If you see objectstore and network numa nodes do not match or not setting numa affinity, the NVMe device and the NIC used by that OSD are attached to different NUMA nodes. See the Discussion below.

Discussion​

Many ARM server processors have a high core count, but lower per-core performance than current x86 server CPUs, and a server typically has several NUMA nodes (check with lscpu or numactl --hardware). With the defaults, an NVMe OSD becomes bottlenecked on a small number of busy threads and on cross-NUMA memory traffic. The options above spread the OSD's work over more cores and keep it on the NUMA node local to its NVMe device and NIC.

tuned network-latency profile​

The network-latency profile builds on latency-performance. Among other things, it:

  • sets the CPU frequency governor to performance and prevents the CPUs from entering deep idle states, which reduces wake-up latency for OSD and messenger threads;
  • disables automatic NUMA balancing (kernel.numa_balancing=0), so the kernel does not migrate memory pages away from the NUMA node that the OSD is pinned to (see osd_numa_auto_affinity below);
  • disables transparent huge pages; and
  • enables busy polling on network sockets.

The cost is higher power consumption, since the CPUs no longer idle in low-power states. How much the frequency and idle-state settings help depends on the cpufreq and cpuidle support of the platform; check with cpupower frequency-info and cpupower idle-info.

ms_async_op_threads​

This is the number of async messenger worker threads per daemon. On ARM servers, the default of 3 threads saturates before the NVMe device does. Raising it to 5 spreads network processing over more cores. The value is read only at messenger start-up, which is why it must be set in ceph.conf via set-extra-ceph-conf and requires an OSD restart.

osd_numa_auto_affinity​

When enabled (the default), the OSD detects the NUMA node of its BlueStore device and of its public and cluster network interfaces. If they all match, the OSD pins itself to the CPUs of that node. This is critical on multi-NUMA ARM servers, where cross-NUMA memory access carries a large penalty.

Auto affinity only applies when storage and network are on the same NUMA node. It is not applied if:

  • the NVMe device sits in a PCIe slot attached to a different NUMA node than the NIC;
  • the public and cluster networks use NICs on different NUMA nodes; or
  • the OSD uses a bond whose members are on different NUMA nodes.

In these cases, fix the physical layout (move NVMe devices or NICs so that each NUMA node has local storage and network), or pin OSDs manually with osd_numa_node:

# ceph config set osd.12 osd_numa_node 2

Use lstopo or cat /sys/class/nvme/nvme0/device/numa_node and cat /sys/class/net/<iface>/device/numa_node to find where each device is attached.

osd_op_queue = wpq​

The default mclock_scheduler limits client IO based on a per-OSD IOPS capacity measured at OSD start-up. On ARM NVMe OSDs this measured capacity is not reliable, which can throttle client IO. The wpq scheduler has no such limits and gives more predictable performance. Note that with wpq, recovery and backfill are controlled by the classic options (osd_max_backfills, osd_recovery_max_active, osd_recovery_sleep_ssd) rather than by the mclock profiles.

osd_memory_target = 20G​

A larger BlueStore cache reduces reads of onode and RocksDB metadata from the NVMe device, which saves CPU time on cores with lower per-core performance. Make sure the host has enough RAM for this: a server with 12 OSDs needs at least 12 * 20G * 1.25 = 300G of RAM for the OSDs alone. See OSD RAM Allocation Recommendations.

osd_op_num_threads_per_shard_ssd = 4​

Each OSD has osd_op_num_shards_ssd (default 8) op shards, each served by osd_op_num_threads_per_shard_ssd (default 2) threads. Raising it to 4 doubles the number of worker threads per OSD, from 16 to 32. This uses the high core count of ARM servers to compensate for their lower per-core speed.

Take the total thread count into account when sizing: with 12 OSDs per host, this means 384 OSD op threads, plus messenger, BlueStore and RocksDB threads.

mon_target_pg_per_osd = 250​

The PG autoscaler sizes pools to reach mon_target_pg_per_osd PGs per OSD. More PGs per OSD gives more parallelism in the OSD op shards and a more even data distribution across OSDs. The cost is higher OSD memory use and longer peering, which is acceptable with a 20G memory target.

mon_max_pg_per_osd must stay well above the target, otherwise PGs may get stuck activating when OSDs fail or during CRUSH changes (see PGs Stuck Activating). If it is lower than about twice the target, raise it:

# ceph config set global mon_max_pg_per_osd 500

Changing mon_target_pg_per_osd on an existing cluster causes the autoscaler to split PGs in the affected pools, which moves data. On a cluster that already holds data, plan the change for a low-traffic period.

Adapting to Other ARM Platforms​

The values in this article are not tuned for any particular ARM processor. When applying them to your platform, consider:

  • Single-NUMA-node servers (for example, a one-socket server where the whole socket is one NUMA node): osd_numa_auto_affinity has no effect, and the NUMA verification steps can be skipped.
  • Fewer cores or hardware threads: the thread counts above may oversubscribe the CPU. Keep the total number of OSD op, messenger and BlueStore threads per host in proportion to the available hardware threads, and reduce ms_async_op_threads and osd_op_num_threads_per_shard_ssd if CPU usage is saturated.
  • Less RAM per OSD: reduce osd_memory_target and mon_target_pg_per_osd together.
  • Virtual machines and cloud instances: the guest may not see the real NUMA topology of the NVMe devices and NICs, so pinning cannot be verified with the steps above.

References​