Skip to main content

Ceph Squid (v19)

This document lists known critical bugs affecting Ceph Squid (v19) releases. Find your version in the first table, then read the details of each bug below.

Bugs by Version​

VersionKnown bugs
Every Squid releaseStretch mode PGs inactive, Zeros from clone snapshot after flatten (reported on 19.2.4), cephadm crash directory ownership (reported on 19.2.3)
19.2.7PGs stuck peering after reboots, OSD key rotation restarts, Monitor cluster log replay
19.2.6PGs stuck peering after reboots, OSD key rotation restarts, OSD missing DB device, Monitor cluster log replay
19.2.4, 19.2.5OSD missing DB device, Monitor cluster log replay
19.2.3New OSDs crash or corrupt data, RGW --bypass-gc data loss, DeleteBucketLifecycle, OSD missing DB device, Monitor cluster log replay
19.2.0, 19.2.1, 19.2.2New OSDs crash or corrupt data, RGW --bypass-gc data loss

Summary​

BugSeverityAffectedFixed in
PGs stuck in activating or peering after OSD restartsHigh19.2.6, 19.2.7Not yet released
Monitors replay months of cluster log after a restartHigh19.2.3 to 19.2.7Not yet released
Cephadm OSD key rotation restarts OSDsCritical19.2.6, 19.2.7Not yet released (fix merged for the next Squid release)
Stretch mode PGs inactive with device-class CRUSH rulesHighAllNot yet released
Reading from clone snapshot after flattening returns zerosReported on 19.2.4Not yet released
cephadm changes ownership of crash directoriesMajorReported on 19.2.3Not in any Squid release
New or recreated OSD missing DB device with hybrid specMedium19.2.3 to 19.2.619.2.7
Squid-deployed OSDs crash, possible data corruptionCriticalOSDs created on 19.2.0 to 19.2.319.2.4 (prevents new corruption only)
RadosGW --bypass-gc data lossCritical19.2.0 to 19.2.319.2.4
S3 DeleteBucketLifecycle does not delete configMedium19.2.319.2.4

PGs Stuck in Activating or Peering After OSD Restarts​

Severity: High Affected Versions: 19.2.6, 19.2.7 (19.2.5 and earlier are not affected) Bug Report: https://tracker.ceph.com/issues/80327

Description​

When OSDs restart, for example during a rolling reboot of the hosts in one rack, some PGs stay in activating or peering for minutes. The primary never hears back from an OSD that has just rejoined, and nothing retries, so the PG stays stuck until the next OSD map change or until an operator re-peers it. Client I/O to those PGs blocks meanwhile.

The cause is a messenger change ("Speed up connection logic when peer restarts") that came into 19.2.6 together with the fix for CVE-2025-30156.

Note: This bug also affects Tentacle 20.2.4. Tentacle 20.2.5 has the fix.

Recommendation​

  • ceph pg repeer <pgid> unblocks a stuck PG immediately. Check for PGs stuck in activating or peering after each host restart.
  • The fix reverts the messenger change. It is in Tentacle 20.2.5; the Squid revert (PR #71956) is under review upstream. Contact Clyso support for a CES build with the fix

Cephadm OSD Key Rotation Bug Causes Unnecessary OSD Restarts​

Severity: Critical Affected Versions: v19.2.6, v19.2.7, v20.2.4 Bug Report: https://tracker.ceph.com/issues/80440

Description​

When a user or cephadm issues an ceph orch daemon reconfig osd.X command on an OSD, it triggers an unnecessary restart of that OSD. The issue is triggered automatically if the manager communicates a new monmap to the OSDs. An example of this is adding a monitor, which triggers a reconfig operation on an OSD. Such a reconfig operation is normal behavior in all versions of Ceph, but added logic in this (the affected) version causes the OSD always to restart.

On an active cluster, this can lead to multiple OSDs starting randomly, and that can cause PGs to become inactive. Sometimes this occurs during a MGR failover. This is always triggered when adding or removing a monitor.

This can be reproduced manually by running the following command:

ceph orch daemon reconfig osd.2 # restarts osd.2, which it shouldn't do

If this has happened in your cluster, it will show up in the cephadm.log of the host's OSD, or in logs that are returned by the command ceph -W cephadm. The following excerpt shows the text that confirms that your OSD key has been rotated:

[DBG] err: Reconfig daemon osd.2 ...
Stopping osd.2 to update osd_key bluestore label
Rotating osd.2 key with ceph-bluestore-tool
Successfully rotated osd.2 keyring

Recommendation​

  • Be aware that mgr failover, monitor add/remove operations, and post-upgrade key rotation can trigger unexpected OSD restarts — plan maintenance windows accordingly.
  • Watch the manager logs for the "Reconfig daemon" / "Rotating ... key" sequence shown above to confirm whether an OSD is being affected by this bug.
  • The fix is merged upstream in PR #71640; the Squid backport (PR #72034) has been merged for the next Squid release (no released version has it yet) and the Tentacle backport (PR #72033) is under review.

Squid Deployed OSDs Are Crashing​

Severity: Critical
Affected Versions: 19.2.[0-3]
Bug Report: https://tracker.ceph.com/issues/70390

Description​

OSDs created in v19 may crash. The issue affects only newly deployed OSDs using Squid, while previously deployed OSDs run fine. It is likely caused by the Elastic Shared Blob implementation introduced in this PR https://github.com/ceph/ceph/pull/53178, and ceph-bluestore-tool repair cannot fix it. In rare cases, can lead to silent data corruption in the OSD.

Recommendation​

  • For now, run ceph config set osd bluestore_elastic_shared_blobs 0 on any Squid cluster before adding new OSDs
  • If you created any OSDs using versions v19.2.0, v19.2.1, v19.2.2, or v19.2.3 and bluestore_elastic_shared_blobs was enabled (default), we strongly recommend completely recreating those specific OSDs to ensure data integrity.

Important Limitation of the Fix​

Ceph v19.2.4 contains a patch for this issue; however, this fix only prevents new corruptions from occurring. It cannot detect or repair existing silent corruptions on OSDs that were created with elastic shared blobs enabled during the affected release window.

S3 DeleteBucketLifecycle Does Not Delete Config​

Severity: Medium
Affected Versions: 19.2.3
Bug Report: https://tracker.ceph.com/issues/71083

Description​

The S3 DeleteBucketLifecycle API call fails to actually delete the lifecycle configuration from the bucket, causing S3 compatibility issues. This is a regression introduced in squid 19.2.3.

Recommendation​

  • Upgrade to a fixed version when available
  • Avoid using DeleteBucketLifecycle API in affected versions

Critical Notice: Potential OSD Data Corruption (v19.2.0 - v19.2.3, v20.2.0)​

A bug has been identified in Ceph versions v19.2.0 through v19.2.3 and Ceph version v20.2.0 that, in rare cases, can lead to silent data corruption in the OSD. This issue impacts any OSD created on these versions while bluestore_elastic_shared_blobs is enabled (which is the default setting).

For technical details, see Tracker Issue #70390.

Important Limitation of the Fix​

Ceph versions v19.2.4 and v20.2.1 contain a patch for this issue; however, this fix only prevents new corruptions from occurring. It cannot detect or repair existing silent corruptions on OSDs that were created with elastic shared blobs enabled during the affected release window.

  • No Action Needed: If all of your OSDs were created prior to the v19 release series, your cluster is not affected.

  • Action Required: If you created any OSDs using versions v19.2.0, v19.2.1, v19.2.2, v19.2.3, or v20.2.0 and bluestore_elastic_shared_blobs = true was enabled, we strongly recommend completely recreating those specific OSDs to ensure data integrity.

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: 19.2.[0-3]
Bug Report: https://tracker.ceph.com/issues/73348

Description​

A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.

Note: This bug also affects Ceph Quincy (v17) and Reef (v18). See the Quincy known bugs and Reef known bugs pages for details.

Recommendation​

  • Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
  • Fixed in 19.2.4, 18.2.8, 20.2.1
  • If you've used --bypass-gc in the past, audit your buckets for objects with missing data

Reading from clone snapshot after concurrent flattening returns zeros​

Affected Versions: reported on v18.2.8, v19.2.4 and v20.2.1 (Reef, Squid and Tentacle)
Bug Report: https://tracker.ceph.com/issues/74554

Description​

Instead of the expected data, zeros are returned in the following case:

An image with an object map is created and filled with non-zero data. A clone is created of the image. A snapshot is created on the clone. The cloned snapshot is opened and remains open. The clone is flattened. Data is read from the cloned snapshot that was opened before flattening.

It is expected that the data written to the parent image is present when data is read from the cloned snapshot that was opened before the flattening operation. Instead, zeros are returned.

Recommendation​

TBD

New or Recreated OSD Missing DB Device with Hybrid Spec​

Severity: Medium Affected Versions: 19.2.3 to 19.2.6 (fixed in 19.2.7)
Bug Report: https://tracker.ceph.com/issues/72696

Description​

When using a hybrid OSD spec (HDDs for data, SSDs/NVMe for DB devices), newly created or recreated OSDs are deployed without a DB device. The ceph-volume lvm batch command issued by the orchestrator omits the --db-device argument.

The root cause is a regression introduced by the fix for tracker #68576: the ceph_device attribute in ceph-volume inventory JSON output was renamed to ceph_device_lvm. This causes the selector to treat all existing RocksDB volumes as unavailable, so it cannot assign DB devices when recreating OSDs.

Note: This bug also affects Ceph Tentacle (v20). See the Tentacle known bugs page for details.

Recommendation​

cephadm Changes Ownership of Crash Directories When Deploying Monitoring Daemons​

Affected Versions: 19.2.3
Component: cephadm
Severity: Major
PR Link: https://github.com/ceph/ceph/pull/66559

Summary​

When deploying monitoring daemons (e.g., Grafana) to a host that already has Ceph daemons running, cephadm incorrectly changes the ownership of the /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories to match the monitoring daemon's UID/GID. This prevents the ceph-crash daemon and other Ceph daemons from posting crash reports, as they run as user 'ceph' and can no longer access these directories.

Steps to Reproduce​

  1. Deploy a crash daemon to a host:

    ceph orch daemon redeploy crash.<hostname>
  2. Verify correct ownership of crash directories:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Expected output:

    drwx------. 19 ceph ceph 4096 Jan 16 19:46 /var/lib/ceph/<fsid>/
    drwx------. 3 ceph ceph 4096 Jan 13 2024 /var/lib/ceph/<fsid>/crash
    drwx------. 2 ceph ceph 4096 Jan 13 2024 /var/lib/ceph/<fsid>/crash/posted
  3. Deploy a monitoring daemon to the same host:

    ceph orch apply grafana <hostname>
  4. Check directory ownership again:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Actual output:

    drwx------. 20 472 root 4096 Jan 17 12:44 /var/lib/ceph/<fsid>/
    drwx------. 3 472 root 4096 Jan 13 2024 /var/lib/ceph/<fsid>/crash
    drwx------. 2 472 root 4096 Jan 13 2024 /var/lib/ceph/<fsid>/crash/posted

Expected Behavior​

The ownership of /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories should remain as ceph:ceph regardless of which daemons are deployed to the host, as these directories are only used by core Ceph daemons.

Actual Behavior​

When a monitoring daemon is deployed, cephadm changes the ownership of the crash directories to match the monitoring daemon's UID/GID (e.g., 472 for Grafana). This causes the following error in ceph-crash logs:

ERROR:ceph-crash:Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash'

The ceph-crash daemon continues to run and monitor the mount point, but is unable to post crash reports. Additionally, other Ceph daemons running as user 'ceph' are unable to write crash dumps to the directory.

Root Cause Analysis​

Examination of src/cephadm/cephadm.py (main branch and 19.2.0) reveals that for each service deployment, the following call chain occurs:

deploy_daemon() -> create_daemon_dirs() -> make_data_dir() -> make_data_dir_base()

In make_data_dir_base():

data_dir_base = os.path.join(data_dir, fsid)                                
makedirs(data_dir_base, uid, gid, DATA_DIR_MODE)
makedirs(os.path.join(data_dir_base, 'crash'), uid, gid, DATA_DIR_MODE)
makedirs(os.path.join(data_dir_base, 'crash', 'posted'), uid, gid,
DATA_DIR_MODE)

The cephadmlib/file_utils.py:makedirs() function always calls os.chown(dest, uid, gid), regardless of whether the directory already exists.

The UID/GID values are obtained from the daemon container using stat -c '%u %g' <file/dir> in cephadmlib/container_types.py:extract_uid_gid(). The file used for this stat operation is determined per-container (e.g., Grafana's is defined in cephadmlib/daemons/monitoring.py:extract_uid_gid()).

The result: The most recently deployed daemon on a host determines the ownership of the /var/lib/ceph/<fsid>, /var/lib/ceph/<fsid>/crash, and /var/lib/ceph/<fsid>/crash/posted directories.

Proposed Solution​

Since the .../crash and .../crash/posted directories are only mounted into core Ceph containers, the solution should prevent ownership changes when the daemon being deployed is not a core Ceph daemon.

The best approach appears to be:

  1. Add a test in cephadmlib/daemons/ceph.py:ceph_daemons() to identify core Ceph daemons
  2. Modify make_data_dir_base() to use a version of makedirs() that changes ownership only on directory creation
  3. This ensures that deploying a monitoring daemon (e.g., Grafana) first on the host will work, but subsequent deployments do not overwrite the ownership of core Ceph daemon directories

Pseudo-code Implementation​

# file_utils.py:
def makedirs_nochange(dest, uid, gid, mode):
"""Create directory with ownership, but don't change if exists"""
if not os.path.exists(dest):
os.makedirs(dest, mode=mode)
os.chown(dest, uid, gid)
os.chmod(dest, mode) # Apply mode (masked by umask on creation)

Workaround​

Until this issue is fixed, the following workaround can be implemented:

Create a cron job or systemd timer that periodically checks for incorrect permissions in crash directories and corrects them:

# Check for crash dirs owned by root or nobody and fix
chown -R 167:167 /var/lib/ceph/<fsid>/crash/
chown 167:167 /var/lib/ceph/<fsid>

Run this hourly or as needed. Note that UID 167 corresponds to the ceph user in the container.

Additional Notes​

  • This issue affects clusters that have been upgraded to 19.2.3
  • The bug impacts crash reporting functionality, which is critical for diagnosing cluster issues
  • Multiple users have reported being affected by this issue after upgrading
  • A patch is pending review (see Ceph Tracker link above)

Stretch Mode PGs Inactive with Device-Class CRUSH Rules​

Severity: High Affected Versions: all releases up to and including 19.2.7 Bug Report: https://tracker.ceph.com/issues/81216

Description​

In stretch mode, pools whose CRUSH rule uses a device class (step take <site> class <class>) go inactive when a site or the inter-site link is lost, and stay inactive after it returns. Pools on rules without a device class are unaffected. The cluster reports HEALTH_OK until a site is lost.

Note: This bug also affects Reef (v18) and Tentacle (v20), and every earlier release with stretch mode.

Recommendation​

Monitors Replay Months of Cluster Log After a Restart​

Severity: High Affected Versions: 19.2.3 to 19.2.7 Bug Report: https://tracker.ceph.com/issues/81604

Description​

Daemons connected to a peon monitor never get acknowledgements for their cluster log messages, so their queues grow in memory for as long as they run. When a monitor restarts, the daemons resend the whole backlog, and the leader commits months of old log entries again: high monitor CPU, tens of thousands of slow ops, and possible loss of quorum on large clusters.

Note: This bug also affects Reef 18.2.8, Tentacle and Umbrella (21.2.0).

Upgrading from an unaffected version (19.2.0 to 19.2.2, or Reef 18.2.7 and earlier) to an affected one brings the bug in: from then on the queues grow again, and each later monitor restart replays more.

Recommendation​

  • Before restarting monitors, set noscrub/nodeep-scrub and restart the other daemons to empty their queues. Don't change clog_to_monitors at runtime: it can crash OSDs. See Monitors Replay Months of Cluster Log After a Restart
  • Fix merged to main in PR #72555; backports under review. Contact Clyso support for a CES build with the fix