Skip to main content

Monitors Replay Months of Cluster Log After a Restart

After a monitor restarts, for example for a kernel update, the monitors process cluster log entries that are weeks or months old. The leader runs at several hundred percent CPU, stops answering clients and OSDs in time, and slow ops on the monitors grow into the tens of thousands. Large clusters can lose quorum.

Affected versions: 18.2.8, 19.2.3 to 19.2.7, every Tentacle release (20.2.0 to 20.2.5), and 21.2.0. 18.2.7 and 19.2.0 to 19.2.2 are not affected.
Bug report: https://tracker.ceph.com/issues/81604 (also https://tracker.ceph.com/issues/81511)
Fix: ceph/ceph PR #72555, merged to main on 4 November 2026. Backports to Squid, Tentacle and Umbrella are under review upstream. There will be no further Reef release.

Cause​

Every Ceph daemon sends its cluster log lines (the ceph -w messages, such as "deep-scrub starts") to its monitor, and keeps each line in memory until the monitors acknowledge it. In the affected versions, a daemon whose session is on a peon monitor never receives that acknowledgement. Its queue of unacknowledged log lines then grows for as long as the daemon runs.

When a monitor restarts, the daemons connected to it resend their whole queue, 1000 entries per message. The monitors only check the last 50 entries for duplicates, so the leader commits months of old entries to the monitor store again.

Scrub messages make up most of the queue. On Reef, OSDs also log "deep-scrub starts" again on every failed scrub reservation attempt, every few seconds.

Check whether you are affected​

Every cluster with more than one monitor that runs an affected version is affected:

ceph versions

The backlog grows with daemon uptime and with the rate of cluster log messages. Daemons that have run for weeks or months since the last monitor restart carry the largest queues.

To see one daemon's backlog, raise its monitor client debug level for a few seconds. Debug levels are safe to change at runtime:

ceph tell osd.<id> config set debug_monc 10
# wait a few seconds, then check the OSD's log for: log_queue is <N>
ceph tell osd.<id> config set debug_monc 1/5

<N> is the number of queued entries. The line appears when the daemon has new entries to send. On an affected daemon it keeps growing; on a healthy one it stays small.

During a replay, the monitors' ops show log(1000 entries from seq ... at <date>) with dates long before the restart:

ceph tell mon.<id> ops | grep 'log(1000 entries'

Before restarting a monitor​

The queues live only in the daemons' memory, so restarting a daemon empties its queue. Before any planned monitor restart, reboot or upgrade:

  1. Stop new scrubs, which produce most of the new entries:

    ceph osd set noscrub
    ceph osd set nodeep-scrub

  2. Restart the other daemons to empty their queues: rolling OSD restarts with noout set, one host at a time; ceph mgr fail; MDS failovers; and RGW, rbd-mirror and other long-running daemons.

  3. Soon after, restart the monitors one at a time, as usual.

  4. Unset the scrub flags:

    ceph osd unset noscrub
    ceph osd unset nodeep-scrub

If a replay has already started​

Don't keep restarting monitors: each restart makes every daemon connected to that monitor replay its whole queue again. Instead, set noscrub and nodeep-scrub, and restart the daemons that are replaying. They are the sources of the log(1000 entries ...) ops on the monitors.

Settings to avoid​

  • Don't change clog_to_monitors at runtime, to any value. On the affected releases a runtime change can crash OSDs (tracker 79444), and ceph config set applies it to every running OSD at once.
  • mon_client_log_interval and mon_client_max_log_entries_per_message can't be changed on running daemons, so they don't slow down a replay.
  • Lowering mon_cluster_log_level doesn't help: it changes what the monitors write, not what the daemons queue.

Recommendation​

  • Contact Clyso support for a CES build with the fix. The fix is in the monitors only: run it on all monitors. Rolling the monitors onto the fixed build still makes the daemons replay their queues once, so empty the queues first, as above.
  • Upgrading to another released upstream version doesn't help yet: no released version has the fix.
  • If you build your own packages, the fix is a small change to src/mon/LogMonitor.cc; see the PR above.