Skip to main content

Resolving CephMonClockSkew Alerts

Problem​

The Ceph cluster occasionally raises a CephMonClockSkew alert. The alert is caused by clock skew exceeding the 0.05s threshold on one or more monitor (mon) nodes.

Let us consider a specific case: The root cause is that ntpd on the Ceph machines synchronises to a particular firewall but has settled on a certain poll interval (1024 seconds, or 17 minutes). Because sntp-command-line-tool queries show near-zero skew on demand, the NTP source itself is considered good--the issue is simply that corrections arrive only every ~17 minutes, and a node's clock can drift beyond the 0.05s threshold between corrections.

The alert self-resolves within approximately 8.5 minutes on average (half the 17-minute poll cycle), without any intervention.

Why This Is Happening​

Several factors contribute to NTP's reduced accuracy:

  • MONs and OSDs are collocated on the same hosts
  • Hosts are under high CPU load due to ongoing recovery and backfilling operations, and high iowait exists due to drives running close to 100% utilisation
  • Network load is also hindering NTP-p=protocol accuracy

The poll interval corresponds to the default maxpoll value of 10 (2¹⁰ = 1024 seconds, see man ntp.conf). The Ceph project uses chrony as its default clock synchronisation daemon because chrony generally performs better than NTP, especially when hardware (CPU and network) is under heavy load.

Risk​

If CephMonClockSkew is not resolved, it could lead to a loss of MON quorum and blocked IOs in worst-case scenarios.

Checking Alert Status​

Run the following on any Ceph MON node to see current clock skew across all MON nodes:

ceph time-sync-status

A node is healthy when its "health" field reads "HEALTH_OK". A node is in violation when "health" reads "HEALTH_WARN" and "details" contains a message such as:

clock skew 0.0511251s > max 0.05s

Monitoring Alert Frequency​

There is currently no Ceph-native archive of historical CephMonClockSkew alerts, and system logs retain only a few hours of activity.

To capture recurrences, periodically download log entries containing time_skew_status and record:

  • The timestamp of each occurrence
  • Which node(s) were affected
  • The skew value reported

This data will help establish how frequently the alert occurs and whether it is worsening over time.

It is possible to check how many times MON_CLOCK_SKEW was triggered and when the last time it was triggered by running the following command:

ceph healthcheck history ls | grep MON_CLOCK_SKEW

Output will look like this:

MON_CLOCK_SKEW            2025/06/19 11:37:24   2026/04/03 10:33:31      15    No

Possible Solutions​

Short-term: Tighten the NTP polling interval​

To reduce the time between clock corrections, adjust the polling interval in ntpd:

  1. Edit /etc/ntp.conf
  2. Adjust the minpoll and maxpoll values on the server line: 
    server <ntp-server> minpoll 4 maxpoll 6
  3. Restart the NTP daemon: 
    systemctl restart ntp

Long-term: Replace ntpd with chrony​

Replacing ntpd with chrony is the preferred long-term solution. chrony is better suited to environments where hardware is under heavy load. The transition can be performed live without any impact on the system.

For further reading, see the Chrony comparison page (https://chrony-project.org/comparison.html), particularly the Performance section.

Before acting on any of these options, confirm that the chosen approach will not cause a step change in clock time on any MON node during the transition, and that all MON node clocks will remain within 50ms of each other throughout.

See also The Ceph Time Synchronization With Chrony page in this knowledge base.