Skip to main content

Ceph Squid (v19)

This document lists known critical bugs affecting Ceph Squid (v19) releases.

Squid Deployed OSDs Are Crashing​

Severity: Critical
Affected Versions: 19.2.[0-3]
Bug Report: https://tracker.ceph.com/issues/70390

Description​

OSDs created in v19 may crash. The issue affects only newly deployed OSDs using Squid, while previously deployed OSDs run fine. It is likely caused by the Elastic Shared Blob implementation introduced in this PR https://github.com/ceph/ceph/pull/53178, and ceph-bluestore-tool repair cannot fix it. In rare cases, can lead to silent data corruption in the OSD.

Recommendation​

  • For now, run ceph config set osd bluestore_elastic_shared_blobs 0 on any Squid cluster before adding new OSDs
  • If you created any OSDs using versions v19.2.0, v19.2.1, v19.2.2, or v19.2.3 and bluestore_elastic_shared_blobs was enabled (default), we strongly recommend completely recreating those specific OSDs to ensure data integrity.

Important Limitation of the Fix​

Ceph v19.2.4 contains a patch for this issue; however, this fix only prevents new corruptions from occurring. It cannot detect or repair existing silent corruptions on OSDs that were created with elastic shared blobs enabled during the affected release window.

S3 DeleteBucketLifecycle Does Not Delete Config​

Severity: Medium
Affected Versions: 19.2.3
Bug Report: https://tracker.ceph.com/issues/71083

Description​

The S3 DeleteBucketLifecycle API call fails to actually delete the lifecycle configuration from the bucket, causing S3 compatibility issues. This is a regression introduced in squid 19.2.3.

Recommendation​

  • Upgrade to a fixed version when available
  • Avoid using DeleteBucketLifecycle API in affected versions

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: 19.2.[0-3]
Bug Report: https://tracker.ceph.com/issues/73348

Description​

A long-standing data loss bug with --bypass-gc causes deletion of copied object data. If any of the deleted objects had been copied to/from other buckets, --bypass-gc deletes the data of those copies too. As a result, the copies are still visible to ListObjects requests but GetObject requests fail with NoSuchKey.

Note: This bug also affects Ceph Quincy (v17) and Reef (v18). See the Quincy known bugs and Reef known bugs pages for details.

Recommendation​

  • Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
  • Fixed in 18.2.8, 20.2.1
  • If you've used --bypass-gc in the past, audit your buckets for objects with missing data

Reading from clone snapshot after concurrent flattening returns zeros​

Bug Report: https://tracker.ceph.com/issues/74554

Description​

Instead of the expected data, zeros are returned in the following case:

An image with an object map is created and filled with non-zero data. A clone is created of the image. A snapshot is created on the clone. The cloned snapshot is opened and remains open. The clone is flattened. Data is read from the cloned snapshot that was opened before flattening.

It is expected that the data written to the parent image is present when data is read from the cloned snapshot that was opened before the flattening operation. Instead, zeros are returned.

Recommendation​

TBD

New or Recreated OSD Missing DB Device with Hybrid Spec​

Severity: Medium Affected Versions: 19.2.3
Bug Report: https://tracker.ceph.com/issues/72696

Description​

When using a hybrid OSD spec (HDDs for data, SSDs/NVMe for DB devices), newly created or recreated OSDs are deployed without a DB device. The ceph-volume lvm batch command issued by the orchestrator omits the --db-device argument.

The root cause is a regression introduced by the fix for tracker #68576: the ceph_device attribute in ceph-volume inventory JSON output was renamed to ceph_device_lvm. This causes the selector to treat all existing RocksDB volumes as unavailable, so it cannot assign DB devices when recreating OSDs.

Note: This bug also affects Ceph Tentacle (v20). See the Tentacle known bugs page for details.

Recommendation​

  • A patch is available in PR #65986
  • Follow the bug tracker for fix and backport updates

cephadm Changes Ownership of Crash Directories When Deploying Monitoring Daemons​

Ceph Version: 19.2.3
Component: cephadm
Severity: Major
PR Link: https://github.com/ceph/ceph/pull/66559

Summary​

When deploying monitoring daemons (e.g., Grafana) to a host that already has Ceph daemons running, cephadm incorrectly changes the ownership of the /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories to match the monitoring daemon's UID/GID. This prevents the ceph-crash daemon and other Ceph daemons from posting crash reports, as they run as user 'ceph' and can no longer access these directories.

Steps to Reproduce​

  1. Deploy a crash daemon to a host:

    ceph orch daemon redeploy crash.<hostname>

  2. Verify correct ownership of crash directories:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Expected output:

    drwx------. 19 ceph ceph 4096 Jan 16 19:46 /var/lib/ceph/<fsid>/
    drwx------.  3 ceph ceph 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash
    drwx------.  2 ceph ceph 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash/posted

  3. Deploy a monitoring daemon to the same host:

    ceph orch apply grafana <hostname>

  4. Check directory ownership again:

    ls -ld /var/lib/ceph/<fsid>/{,crash,crash/posted}

    Actual output:

    drwx------. 20 472 root 4096 Jan 17 12:44 /var/lib/ceph/<fsid>/
    drwx------.  3 472 root 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash
    drwx------.  2 472 root 4096 Jan 13  2024 /var/lib/ceph/<fsid>/crash/posted

Expected Behavior​

The ownership of /var/lib/ceph/<fsid>/crash and /var/lib/ceph/<fsid>/crash/posted directories should remain as ceph:ceph regardless of which daemons are deployed to the host, as these directories are only used by core Ceph daemons.

Actual Behavior​

When a monitoring daemon is deployed, cephadm changes the ownership of the crash directories to match the monitoring daemon's UID/GID (e.g., 472 for Grafana). This causes the following error in ceph-crash logs:

ERROR:ceph-crash:Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash'

The ceph-crash daemon continues to run and monitor the mount point, but is unable to post crash reports. Additionally, other Ceph daemons running as user 'ceph' are unable to write crash dumps to the directory.

Root Cause Analysis​

Examination of src/cephadm/cephadm.py (main branch and 19.2.0) reveals that for each service deployment, the following call chain occurs:

deploy_daemon() -> create_daemon_dirs() -> make_data_dir() -> make_data_dir_base()

In make_data_dir_base():

data_dir_base = os.path.join(data_dir, fsid)                                
makedirs(data_dir_base, uid, gid, DATA_DIR_MODE)                            
makedirs(os.path.join(data_dir_base, 'crash'), uid, gid, DATA_DIR_MODE)     
makedirs(os.path.join(data_dir_base, 'crash', 'posted'), uid, gid,          
         DATA_DIR_MODE)

The cephadmlib/file_utils.py:makedirs() function always calls os.chown(dest, uid, gid), regardless of whether the directory already exists.

The UID/GID values are obtained from the daemon container using stat -c '%u %g' <file/dir> in cephadmlib/container_types.py:extract_uid_gid(). The file used for this stat operation is determined per-container (e.g., Grafana's is defined in cephadmlib/daemons/monitoring.py:extract_uid_gid()).

The result: The most recently deployed daemon on a host determines the ownership of the /var/lib/ceph/<fsid>, /var/lib/ceph/<fsid>/crash, and /var/lib/ceph/<fsid>/crash/posted directories.

Proposed Solution​

Since the .../crash and .../crash/posted directories are only mounted into core Ceph containers, the solution should prevent ownership changes when the daemon being deployed is not a core Ceph daemon.

The best approach appears to be:

  1. Add a test in cephadmlib/daemons/ceph.py:ceph_daemons() to identify core Ceph daemons
  2. Modify make_data_dir_base() to use a version of makedirs() that changes ownership only on directory creation
  3. This ensures that deploying a monitoring daemon (e.g., Grafana) first on the host will work, but subsequent deployments do not overwrite the ownership of core Ceph daemon directories

Pseudo-code Implementation​

# file_utils.py:
def makedirs_nochange(dest, uid, gid, mode):
    """Create directory with ownership, but don't change if exists"""
    if not os.path.exists(dest):
        os.makedirs(dest, mode=mode)
        os.chown(dest, uid, gid)
    os.chmod(dest, mode)  # Apply mode (masked by umask on creation)

Workaround​

Until this issue is fixed, the following workaround can be implemented:

Create a cron job or systemd timer that periodically checks for incorrect permissions in crash directories and corrects them:

# Check for crash dirs owned by root or nobody and fix
chown -R 167:167 /var/lib/ceph/<fsid>/crash/
chown 167:167 /var/lib/ceph/<fsid>

Run this hourly or as needed. Note that UID 167 corresponds to the ceph user in the container.

Additional Notes​

  • This issue affects clusters that have been upgraded to 19.2.3
  • The bug impacts crash reporting functionality, which is critical for diagnosing cluster issues
  • Multiple users have reported being affected by this issue after upgrading
  • A patch is pending review (see Ceph Tracker link above)