Skip to main content

OSD RAM Allocation Recommendations

Overview​

RAM allocation for Ceph OSDs depends on several factors including drive size, deployment type, and workload characteristics. This article provides practical guidelines for determining appropriate RAM allocation per OSD.

Official Recommendation​

According to the Ceph documentation, the recommended approach is to ensure that total server RAM is greater than:

(number of OSDs * osd_memory_target * 2)

This calculation allows for:

  • RAM usage by the operating system
  • RAM usage by other Ceph daemons

Reference: https://docs.ceph.com/en/latest/start/hardware-recommendations/

Pragmatic Heuristics​

There are various opinions and methods for calculating OSD RAM allocation. A pragmatic heuristic approach depends on whether the server is dedicated to OSDs or running other services.

Dedicated OSD Servers​

For servers dedicated to running only OSDs:

RAM per OSD = (Total RAM * 0.8) / Number of OSDs

This formula reserves 20% of total RAM for the OS and overhead, distributing the remaining 80% across all OSDs.

Mixed-Use Servers​

For servers also running other Ceph services (such as RGW):

RAM per OSD = (Total RAM - Expected RAM for other services) / Number of OSDs

Subtract the expected RAM usage for other services from the total RAM before dividing by the number of OSDs.

Minimum and Maximum Guidelines​

Minimum RAM per OSD​

4 GB minimum is recommended for most deployments.

Exception: Extreme low-cost or archive-only deployments may use less than 4 GB, but this should only be done with careful consideration of the workload and performance requirements.

Maximum RAM per OSD​

12-16 GB maximum is a practical upper limit.

There are diminishing returns above approximately 8 GB per OSD. While the exact maximum is not well established, allocating more than 12-16 GB per OSD typically provides minimal additional benefit.

Workload-Dependent Considerations​

RAM requirements depend heavily on the number of active onodes in the current global working dataset and the desired onode hit rate.

Example Workload Scenarios​

High Random Access Workload: A cluster full of 4K RGW objects with thousands of users randomly accessing objects across OSDs will have very different RAM requirements compared to other workloads.

Sequential I/O Workload: A cluster with a couple of users performing sequential I/O against a modest unfragmented dataset will require less RAM for similar performance.

Drive Size Considerations​

Generally speaking, lower ratios of memory to TB of disk space are acceptable at larger drive sizes. As drive capacity increases, the RAM-to-capacity ratio can decrease while maintaining acceptable performance.

Summary​

  • Minimum: 4 GB per OSD (except extreme cases)
  • Maximum: 12-16 GB per OSD (diminishing returns above ~8 GB)
  • Dedicated OSD servers: (Total RAM * 0.8) / Number of OSDs
  • Mixed-use servers: (Total RAM - RAM for other services) / Number of OSDs
  • Official formula: Total RAM > (Number of OSDs * osd_memory_target * 2)

The appropriate RAM allocation ultimately depends on your specific workload characteristics, drive sizes, and performance requirements.