Skip to main content

Ceph Umbrella (v21)

This document lists known critical bugs affecting Ceph Umbrella (v21) releases. Find your version in the first table, then read the details of each bug below.

Bugs by Version​

VersionKnown bugs
Every Umbrella releaseMonitor cluster log replay
21.2.0BlueFS overwrite by deferred replay, Optimized EC returns zeros, Cephadm uppercase hostnames, RGW multisite upgrade inconsistency, LRC EIO after upgrade, OSD start timeout during upgrade, Active MGR memory leak

Summary​

BugSeverityAffectedFixed in
Monitors replay months of cluster log after a restartHigh21.2.0Not yet released
BlueFS overwritten by deferred replayCriticalAllNot yet released
Optimized EC returns zeros after unaligned writesCritical21.2.0, pools with EC optimizationsNot yet released
RGW multisite inconsistency during rolling upgradeHigh21.2.0 multisiteNot yet released
Cephadm marks uppercase hostnames as unknown after upgradeHighUpgrades to 21.2.0Not yet released
LRC pools return EIO on degraded reads after upgradeHighLRC pools created before TentacleNot yet released
OSDs hit the start timeout during upgradeMediumUpgrades under loadNot yet released
Active MGR memory leakMedium21.2.0Not yet released

Bugs that link to the Tentacle page are described there.

BlueFS Overwritten by Deferred Replay​

Severity: Critical Affected Versions: All Bug Report: https://tracker.ceph.com/issues/82617

Description​

During OSD startup, _deferred_replay can write over BlueFS extents and corrupt the OSD's RocksDB. It can hit on the first OSD start after an upgrade.

Recommendation​

  • The Umbrella backport is PR #73848 (under review). No released version has it yet.

Optimized EC Returns Zeros After Unaligned Writes​

Severity: Critical Affected Versions: 21.2.0, only on pools with allow_ec_optimizations enabled Bug Report: https://tracker.ceph.com/issues/81848

Description​

With EC optimizations enabled, two unaligned writes that share a page can leave a gap that reads back as zeros. PGs stay clean and deep scrub does not detect it.

Recommendation​

  • Do not enable allow_ec_optimizations on 21.2.0.
  • The Umbrella backport is PR #73835 (under review).

RGW Multisite Inconsistency During Rolling Upgrade​

Severity: High Affected Versions: 21.2.0 multisite, upgrades from Squid or Tentacle Bug Report: https://tracker.ceph.com/issues/82574, https://tracker.ceph.com/issues/82573

Description​

During a rolling multisite upgrade, stale version-list keys are never removed. On Umbrella, a stale null delete marker arriving through sync can delete the data of the current null version.

Recommendation​

  • Delay Umbrella upgrades of RGW multisite deployments until a fix is released.

Cephadm Marks Uppercase Hostnames as Unknown After Upgrade​

Severity: High Affected Versions: Upgrades from 20.2.x to 21.2.0 Bug Report: https://tracker.ceph.com/issues/82666

Description​

After the upgrade, hosts with uppercase characters in their names show up as "Unknown hosts" and the orchestrator becomes unusable.

Recommendation​

  • Check ceph orch host ls for uppercase hostnames before upgrading. If any exist, delay the upgrade.

LRC Pools Return EIO on Degraded Reads After Upgrade​

Severity: High Affected Versions: 21.2.0, LRC pools created before Tentacle Bug Report: https://tracker.ceph.com/issues/81369

Description​

Degraded reads from LRC pools created on Squid or earlier fail with EIO after upgrading.

Recommendation​

  • Clusters with LRC pools should not upgrade until a fix is released.

OSDs Hit the Start Timeout During Upgrade​

Severity: Medium Affected Versions: Upgrades to 21.2.0 under client load Bug Report: https://tracker.ceph.com/issues/81933

Description​

Under load, upgraded OSDs can exceed the systemd start timeout (200 s) and are marked failed.

Recommendation​

  • Upgrade during low-load windows and use a staggered ceph orch upgrade.

Active MGR Memory Leak​

Severity: Medium Affected Versions: 21.2.0 Bug Report: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WT7NTYUUL6HMJ5CGHRP6BHRJ3W37JUHN/

Description​

The active MGR grows by about 1 GB per hour.

Recommendation​

  • Workaround: ceph config set mgr mgr_map_cache_enabled false. Slow growth remains, so fail over the MGR periodically.