Skip to main content

Fix an OSD that has become associated with two block devices

Problem

A user replaced some OSDs in a cluster. After the replacement, one of the OSDs in the cluster had two block devices associated with it. The user wanted only one block device to be associated with the OSD.

Solution

  1. Confirm that more than one block device is associated with the OSD under consideration by running the command cephadm ceph-volume lvm list xxx, where xxx is the number associated with the OSD:

    $ cephadm ceph-volume lvm list 001 

    Using recent ceph image registry.suse.com/ses/7.1/ceph/ceph@sha256:000aaaaaa00aa0aa00000aa00a0000000000000aa0a000aa0a00000a0a000a0a

    ceph-00000aa0-000a-0a00-a0a0-aa0a0aaa0000/osd-block-a0a00a00-a000-0a0b-aa00-a00000a000a0

    [block] /dev/ceph-AAAAAaa0-000a-0a00-a0a0-aa0a0aaa0000/osd-block-a0a00a00-a000-0a0a-aa00-a00000a000a0

    block device /dev/ceph-AAAAAaa0-000a-0a00-a0a0-aa0a0aaa0000/osd-block-a0a00a00-a000-0a0a-aa00-a00000a000a0
    block uuid xxxxxx-xxxx-xxx0-0xxx-xxx-xxxx-xxx0xx
    cephx lockbox secret
    cluster fsid xxxxxxxx-xxxx-xxxx-xx00-xxxxxxxxxxxx
    cluster name ceph
    crush device class
    encrypted 0
    osd fsid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    osd id 209
    osdspec affinity drive_group_ssd-backed
    type block
    vdo 0
    devices /dev/sdq

    [block] /dev/ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/osd-block-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

    block device /dev/ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/osd-block-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    block uuid xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx
    cephx lockbox secret
    cluster fsid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    cluster name ceph
    crush device class
    db device /dev/ceph-block-dbs-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/osd-db-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    db uuid xxxxxx-xxmx-xxxx-xxxx-xxxx-xxxx-xxxxxx
    encrypted 0
    osd fsid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    osd id 209
    osdspec affinity drive_group_ssd-backed
    type block
    vdo 0
    devices /dev/sdag
  2. Use the command dmsetup remove to remove the target logical volume:

    dmsetup remove ceph-AAAAAaa0-000a-0a00-a0a0-aa0a0aaa0000/osd-block-a0a00a00-a000-0a0b-aa00-a00000a000a0
  3. Use the command wipefs -a to remove /dev/sdq.

    wipefs -a /dev/sdq

    Warning: Using wipefs deletes the filesystem on the disk (in this example /dev/sdq). This command erases all filesystem, RAID, and partition-table signatures on the storage device (in this example /dev/sdq).

  4. /dev/sdq is now empty and can be removed from the system.