Ceph Reef (v18)
This document lists known critical bugs affecting Ceph Reef (v18) releases. Reef is end of life: 18.2.8 is the last release, so bugs still open will not be fixed in a Reef release. Find your version in the first table, then read the details of each bug below.
Bugs by Version
| Version | Known bugs |
|---|---|
| Every Reef release | Stretch mode PGs inactive |
| 18.2.8 | Monitor cluster log replay, Zeros from clone snapshot after flatten (reported on 18.2.8) |
| 18.2.7 | RGW --bypass-gc data loss |
| 18.2.5, 18.2.6 | BlueStore potential corruption, RGW --bypass-gc data loss |
| 18.2.1 to 18.2.4 | RGW --bypass-gc data loss |
| 18.2.0 | RGW --bypass-gc data loss, RGW seg-faults with TempURL (with 17.2.7 RGW nodes) |
Summary
| Bug | Severity | Affected | Fixed in |
|---|---|---|---|
| Monitors replay months of cluster log after a restart | High | 18.2.8 | No Reef release |
| Stretch mode PGs inactive with device-class CRUSH rules | High | All | No Reef release |
| Reading from clone snapshot after flattening returns zeros | Reported on 18.2.8 | No Reef release | |
| RadosGW --bypass-gc data loss | Critical | 18.2.0 to 18.2.7 | 18.2.8 |
| RGW seg-faults with TempURL | Major | 18.2.0 with 17.2.7 RGW nodes | 18.2.8 |
| BlueStore potential corruption | Critical | 18.2.5, 18.2.6 | 18.2.7 |
The clone snapshot bug is described on the Squid page.
BlueStore Potential Corruption
Severity: Critical
Affected Versions: 18.2.5, 18.2.6
Bug Report: https://tracker.ceph.com/issues/69764
Description
Some versions of Ceph were released with a bug that may cause OSDs to crash and corrupt the on-disk data.
Recommendation
Upgrade to a fixed version (17.2.9 or 18.2.7) as soon as possible.
RadosGW --bypass-gc Data Loss Bug
Severity: Critical
Affected Versions: 18.2.[0-7]
Bug Report: https://tracker.ceph.com/issues/73348
Description
A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.
Note: This bug also affects Ceph Quincy (v17) and Squid (v19). See the Quincy known bugs and Squid known bugs pages for details.
Recommendation
- Use
radosgw-admin bucket rmwithout--bypass-gcwhich correctly handles copied objects - Fixed in 18.2.8
- If you've used
--bypass-gcin the past, audit your buckets for objects with missing data
RGW seg-faults with TempURL
Severity: major Affected Versions: 18.2.0 (with 17.2.7 RGW nodes) Bug Report: https://tracker.ceph.com/issues/63684
RGW daemons can segfault one after another on Swift TempURL requests. See RGW seg-faults with TempURL in Quincy for details. Upgrade to 18.2.8 or higher.
Stretch Mode PGs Inactive with Device-Class CRUSH Rules
Severity: High Affected Versions: all releases up to and including 18.2.8 Bug Report: https://tracker.ceph.com/issues/81216
Description
In stretch mode, pools whose CRUSH rule uses a device class (step take <site> class <class>) go inactive when a site or the inter-site link is lost, and stay inactive after it returns. Pools on rules without a device class are unaffected. The cluster reports HEALTH_OK until a site is lost.
Note: This bug also affects Squid (v19) and Tentacle (v20), and every earlier release with stretch mode.
Recommendation
- Check whether you are affected, and apply the interim workaround (
ceph osd force_healthy_stretch_mode --yes-i-really-mean-itonce the site is back): see Stretch Mode PGs Inactive with Device-Class CRUSH Rules - Fix pending upstream in PR #72252; contact Clyso support for a hotfix build
Monitors Replay Months of Cluster Log After a Restart
Severity: High Affected Versions: 18.2.8 (18.2.7 is not affected) Bug Report: https://tracker.ceph.com/issues/81604
Description
Daemons connected to a peon monitor never get acknowledgements for their cluster log messages, so their queues grow in memory for as long as they run. When a monitor restarts, the daemons resend the whole backlog, and the leader commits months of old log entries again: high monitor CPU, tens of thousands of slow ops, and possible loss of quorum on large clusters.
Note: This bug also affects Squid (19.2.3 and later), Tentacle and Umbrella (21.2.0). There will be no further Reef release with the fix.
Upgrading from 18.2.7 or earlier to 18.2.8 brings the bug in: from then on the queues grow again, and each later monitor restart replays more.
Recommendation
- Before restarting monitors, set
noscrub/nodeep-scruband restart the other daemons to empty their queues. Don't changeclog_to_monitorsat runtime: it can crash OSDs. See Monitors Replay Months of Cluster Log After a Restart - Fix merged to main in PR #72555; backports under review. Contact Clyso support for a CES build with the fix