Skip to main content

Ceph Squid (v19)

This document lists known critical bugs affecting Ceph Squid (v19) releases.

Squid Deployed OSDs Are Crashing​

Severity: Critical
Affected Versions: 19.2.*
Bug Report: https://tracker.ceph.com/issues/70390

Description​

OSDs created in v19 may crash. The issue affects only newly deployed OSDs using Squid, while previously deployed OSDs run fine. It is likely caused by the Elastic Shared Blob implementation introduced in this PR https://github.com/ceph/ceph/pull/53178, and ceph-bluestore-tool repair cannot fix it.

Recommendation​

  • For now, run ceph config set osd bluestore_elastic_shared_blobs 0 on any Squid cluster before adding new OSDs
  • Unfortunately, this will not help OSDs already deployed in Squid—the only known fix for them is redeployment

S3 DeleteBucketLifecycle Does Not Delete Config​

Severity: Medium
Affected Versions: 19.2.3
Bug Report: https://tracker.ceph.com/issues/71083

Description​

The S3 DeleteBucketLifecycle API call fails to actually delete the lifecycle configuration from the bucket, causing S3 compatibility issues. This is a regression introduced in squid 19.2.3.

Recommendation​

  • Upgrade to a fixed version when available
  • Avoid using DeleteBucketLifecycle API in affected versions

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: 19.2.x
Bug Report: https://tracker.ceph.com/issues/73348

Description​

A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.

Note: This bug also affects Ceph Quincy (v17) and Reef (v18). See the Quincy known bugs and Reef known bugs pages for details.

Recommendation​

  • Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
  • Follow the bug tracker for fix and backport updates
  • If you've used --bypass-gc in the past, audit your buckets for objects with missing data

Reading from clone snapshot after concurrent flattening returns zeros​

Bug Report: https://tracker.ceph.com/issues/74554

Description​

Instead of the expected data, zeros are returned in the following case:

An image with an object map is created and filled with non-zero data. A clone is created of the image. A snapshot is created on the clone. The cloned snapshot is opened and remains open. The clone is flattened. Data is read from the cloned snapshot that was opened before flattening.

It is expected that the data written to the parent image is present when data is read from the cloned snapshot that was opened before the flattening operation. Instead, zeros are returned.

Recommendation​

TBD

cephadm Changes Ownership of Crash Directories When Deploying Monitoring Daemons​

Ceph Version: 19.2.3
Component: cephadm
Severity: Major
PR Link: https://github.com/ceph/ceph/pull/66559

Summary​

When deploying monitoring daemons (e.g., Grafana) to a host that already has Ceph daemons running, cephadm incorrectly changes the ownership of the /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories to match the monitoring daemon's UID/GID. This prevents the ceph-crash daemon and other Ceph daemons from posting crash reports, as they run as user 'ceph' and can no longer access these directories.

Steps to Reproduce​

  1. Deploy a crash daemon to a host:

    ceph orch daemon redeploy crash.<hostname>

  2. Verify correct ownership of crash directories:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Expected output:

    drwx------. 19 ceph ceph 4096 Jan 16 19:46 /var/lib/ceph/<fsid>/
    drwx------.  3 ceph ceph 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash
    drwx------.  2 ceph ceph 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash/posted

  3. Deploy a monitoring daemon to the same host:

    ceph orch apply grafana <hostname>

  4. Check directory ownership again:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Actual output:

    drwx------. 20 472 root 4096 Jan 17 12:44 /var/lib/ceph/<fsid>/
    drwx------.  3 472 root 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash
    drwx------.  2 472 root 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash/posted

Expected Behavior​

The ownership of /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories should remain as ceph:ceph regardless of which daemons are deployed to the host, as these directories are only used by core Ceph daemons.

Actual Behavior​

When a monitoring daemon is deployed, cephadm changes the ownership of the crash directories to match the monitoring daemon's UID/GID (e.g., 472 for Grafana). This causes the following error in ceph-crash logs:

ERROR:ceph-crash:Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash'

The ceph-crash daemon continues to run and monitor the mount point, but is unable to post crash reports. Additionally, other Ceph daemons running as user 'ceph' are unable to write crash dumps to the directory.

Root Cause Analysis​

Examination of src/cephadm/cephadm.py (main branch and 19.2.0) reveals that for each service deployment, the following call chain occurs:

deploy_daemon() -> create_daemon_dirs() -> make_data_dir() -> make_data_dir_base()

In make_data_dir_base():

data_dir_base = os.path.join(data_dir, fsid)                                
makedirs(data_dir_base, uid, gid, DATA_DIR_MODE)                            
makedirs(os.path.join(data_dir_base, 'crash'), uid, gid, DATA_DIR_MODE)     
makedirs(os.path.join(data_dir_base, 'crash', 'posted'), uid, gid,          
         DATA_DIR_MODE)

The cephadmlib/file_utils.py:makedirs() function always calls os.chown(dest, uid, gid), regardless of whether the directory already exists.

The UID/GID values are obtained from the daemon container using stat -c '%u %g' <file/dir> in cephadmlib/container_types.py:extract_uid_gid(). The file used for this stat operation is determined per-container (e.g., Grafana's is defined in cephadmlib/daemons/monitoring.py:extract_uid_gid()).

The result: The most recently deployed daemon on a host determines the ownership of the /var/lib/ceph/<fsid>, /var/lib/ceph/<fsid>/crash, and /var/lib/ceph/<fsid>/crash/posted directories.

Proposed Solution​

Since the .../crash and .../crash/posted directories are only mounted into core Ceph containers, the solution should prevent ownership changes when the daemon being deployed is not a core Ceph daemon.

The best approach appears to be:

  1. Add a test in cephadmlib/daemons/ceph.py:ceph_daemons() to identify core Ceph daemons
  2. Modify make_data_dir_base() to use a version of makedirs() that changes ownership only on directory creation
  3. This ensures that deploying a monitoring daemon (e.g., Grafana) first on the host will work, but subsequent deployments do not overwrite the ownership of core Ceph daemon directories

Pseudo-code Implementation​

# file_utils.py:
def makedirs_nochange(dest, uid, gid, mode):
    """Create directory with ownership, but don't change if exists"""
    if not os.path.exists(dest):
        os.makedirs(dest, mode=mode)
        os.chown(dest, uid, gid)
    os.chmod(dest, mode)  # Apply mode (masked by umask on creation)

Workaround​

Until this issue is fixed, the following workaround can be implemented:

Create a cron job or systemd timer that periodically checks for incorrect permissions in crash directories and corrects them:

# Check for crash dirs owned by root or nobody and fix
chown -R 167:167 /var/lib/ceph/<fsid>/crash/
chown 167:167 /var/lib/ceph/<fsid>

Run this hourly or as needed. Note that UID 167 corresponds to the ceph user in the container.

Additional Notes​

  • This issue affects clusters that have been upgraded to 19.2.3
  • The bug impacts crash reporting functionality, which is critical for diagnosing cluster issues
  • Multiple users have reported being affected by this issue after upgrading
  • A patch is pending review (see Ceph Tracker link above)