2 posts tagged with "linux"

During maintenance work, we noticed that a "ceph clock skew detected" always occurred when restarting the Ceph monitors.

What makes this interesting for the Ceph Monitor Leader is the fact that it assumes that its time is correct and that everyone else's time is wrong.

This means that the cluster remains in the "ceph clock skew detected" state with all its consequences until the time is synchronized.

All ceph monitors and osd nodes are synchronized via ntp and so the state lasts for about 20 seconds.

Possible solutions:

Synchronize the hardware clock before rebooting

hwclock --systohc

When starting the system, do not start the Ceph Monitor until the time has been synchronized.

However, you should always bear in mind the risks and side effects on rainy days.

First you need access to the boot screen.

Than reboot your server, you will see the boot loader screen, select the recovery mode.

Type e for edit and add the following option to the kernel boot options:

init=/bin/bash

Press Enter to exit the edit mode and boot into the single-user mode.

This will boot the kernel with /bin/bash instead of the standard init.