Skip to main content

Ceph Reef (v18)

This document lists known critical bugs affecting Ceph Reef (v18) releases. Reef is end of life: 18.2.8 is the last release, so bugs still open will not be fixed in a Reef release. Find your version in the first table, then read the details of each bug below.

Bugs by Version​

VersionKnown bugs
Every Reef releaseStretch mode PGs inactive
18.2.8Monitor cluster log replay, Zeros from clone snapshot after flatten (reported on 18.2.8)
18.2.7RGW --bypass-gc data loss
18.2.5, 18.2.6BlueStore potential corruption, RGW --bypass-gc data loss
18.2.1 to 18.2.4RGW --bypass-gc data loss
18.2.0RGW --bypass-gc data loss, RGW seg-faults with TempURL (with 17.2.7 RGW nodes)

Summary​

BugSeverityAffectedFixed in
Monitors replay months of cluster log after a restartHigh18.2.8No Reef release
Stretch mode PGs inactive with device-class CRUSH rulesHighAllNo Reef release
Reading from clone snapshot after flattening returns zerosReported on 18.2.8No Reef release
RadosGW --bypass-gc data lossCritical18.2.0 to 18.2.718.2.8
RGW seg-faults with TempURLMajor18.2.0 with 17.2.7 RGW nodes18.2.8
BlueStore potential corruptionCritical18.2.5, 18.2.618.2.7

The clone snapshot bug is described on the Squid page.

BlueStore Potential Corruption​

Severity: Critical
Affected Versions: 18.2.5, 18.2.6
Bug Report: https://tracker.ceph.com/issues/69764

Description​

Some versions of Ceph were released with a bug that may cause OSDs to crash and corrupt the on-disk data.

Recommendation​

Upgrade to a fixed version (17.2.9 or 18.2.7) as soon as possible.

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: 18.2.[0-7] Bug Report: https://tracker.ceph.com/issues/73348

Description​

A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.

Note: This bug also affects Ceph Quincy (v17) and Squid (v19). See the Quincy known bugs and Squid known bugs pages for details.

Recommendation​

  • Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
  • Fixed in 18.2.8
  • If you've used --bypass-gc in the past, audit your buckets for objects with missing data

RGW seg-faults with TempURL​

Severity: major Affected Versions: 18.2.0 (with 17.2.7 RGW nodes) Bug Report: https://tracker.ceph.com/issues/63684

RGW daemons can segfault one after another on Swift TempURL requests. See RGW seg-faults with TempURL in Quincy for details. Upgrade to 18.2.8 or higher.

Stretch Mode PGs Inactive with Device-Class CRUSH Rules​

Severity: High Affected Versions: all releases up to and including 18.2.8 Bug Report: https://tracker.ceph.com/issues/81216

Description​

In stretch mode, pools whose CRUSH rule uses a device class (step take <site> class <class>) go inactive when a site or the inter-site link is lost, and stay inactive after it returns. Pools on rules without a device class are unaffected. The cluster reports HEALTH_OK until a site is lost.

Note: This bug also affects Squid (v19) and Tentacle (v20), and every earlier release with stretch mode.

Recommendation​

Monitors Replay Months of Cluster Log After a Restart​

Severity: High Affected Versions: 18.2.8 (18.2.7 is not affected) Bug Report: https://tracker.ceph.com/issues/81604

Description​

Daemons connected to a peon monitor never get acknowledgements for their cluster log messages, so their queues grow in memory for as long as they run. When a monitor restarts, the daemons resend the whole backlog, and the leader commits months of old log entries again: high monitor CPU, tens of thousands of slow ops, and possible loss of quorum on large clusters.

Note: This bug also affects Squid (19.2.3 and later), Tentacle and Umbrella (21.2.0). There will be no further Reef release with the fix.

Upgrading from 18.2.7 or earlier to 18.2.8 brings the bug in: from then on the queues grow again, and each later monitor restart replays more.

Recommendation​

  • Before restarting monitors, set noscrub/nodeep-scrub and restart the other daemons to empty their queues. Don't change clog_to_monitors at runtime: it can crash OSDs. See Monitors Replay Months of Cluster Log After a Restart
  • Fix merged to main in PR #72555; backports under review. Contact Clyso support for a CES build with the fix