Skip to main content

Ceph Quincy (v17)

This document lists known critical bugs affecting Ceph Quincy (v17) releases.

PG Splitting/Merging Causes OSD Out-Of-Memory​

Severity: High
Affected Versions: 17.2.0, 17.2.1, 17.2.2, 17.2.3
Bug Report: https://tracker.ceph.com/issues/53729

Description​

A bug in the PG splitting and merging code can cause the OSD to go out-of-memory, a condition which persists even after restart. Offline tools are available in fixed releases to workaround the issue.

Recommendation​

  • Do not change pg_num for any pool until after upgrade to a fixed release
  • Disable the pg autoscaler
  • Fixed in v17.2.4

BlueStore Potential Corruption​

Severity: Critical
Affected Versions: 17.2.8
Bug Report: https://tracker.ceph.com/issues/69764

Description​

Some versions of Ceph were released with a bug that may cause OSDs to crash and corrupt the on-disk data.

Recommendation​

Upgrade to a fixed version (17.2.9 or 18.2.7) as soon as possible.

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: 17.2.x
Bug Report: https://tracker.ceph.com/issues/73348

Description​

A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.

Note: This bug also affects Ceph Reef (v18) and Squid (v19). See the Reef known bugs and Squid known bugs pages for details.

Recommendation​

  • Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
  • Fixed in 18.2.8, 20.2.1
  • If you've used --bypass-gc in the past, audit your buckets for objects with missing data

After a host reboot, cephadm refuses to exit maintenance mode​

Severity: minor Affected Versions: 17.2.7 Bug Report: https://tracker.ceph.com/issues/67905

Description​

Under certain circumstances, SSH throws an error to the user and drops its connection.

For example, when attempting to use cephadm to exit maintenance mode, you might see this:

#Exiting from maintenance mode - Failed
root@mcw-ceph017-04:/# ceph orch host maintenance exit mcw-ceph018-32
Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1757, in _handle_command
  return self.handle_command(inbuf, cmd)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command
  return dispatch[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
  return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda>
  wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)  # noqa: E731
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper
  return func(*args, **kwargs)
File "/usr/share/ceph/mgr/orchestrator/module.py", line 575, in _host_maintenance_exit
  raise_if_exception(completion)
File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception
  e = pickle.loads(c.serialized_exception)
TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr'

Recommendation​

RGW seg-faults with TempURL in Quincy​

Severity: major Affected Versions: 18.2.0 (with 17.2.7 RGW nodes) Bug Report: https://tracker.ceph.com/issues/63684

Description​

Under certain circumstances, when a Swift get_obj() call attempts to read object permissions via RADOS, RGW segfaults.

This issue was observed in Ceph deployment containing three RGW daemons. When the first RGW daemon hit the contition that caused the segfault, the request that provoked the segfault is retried on a second daemon. That second daemon segfault, and the process is repeated on the third RGW daemon. You guessed it: the third RGW daemon segfaults.

Recommendation​

Upgrade to 18.2.8 or higher, the final version of v18.

See Also​

https://tracker.ceph.com/issues/67825 https://tracker.ceph.com/issues/63684 https://github.com/ceph/ceph/pull/52433 https://github.com/ceph/ceph/pull/39876 https://github.com/ceph/ceph/pull/56004