Definition

PG splitting with upmap-remapped is a technique used to speed up the PG splitting process in Ceph clusters by preventing unnecessary data movement during splits.

How It Works

When pg_num is increased, PGs are split. When splitting a big data pool, the cluster will need to first resolve misplaced objects before further increase to pg_num can occur.

This flow looks like outlined below:

pgp_num gets increased -> creates misplaced objects -> wait for everything to rebalance -> pgp_num gets increased

Waiting for rebalancing to happen during splitting PGs for a big data pool can last hours or days. Using upmap-remapped script can deffer balancing to once all PGs have been split.

pgp_num gets increased -> creates misplaced objects -> ./upmap-remapped removes misplaced objects -> pgp_num gets increased

Once this finished and all new PGs have been created, rebalancing can be done after.

# turn off balancing
ceph balancer off
ceph osd set norebalance

# increase the pg numbers
ceph osd pool set <pool> pg_num <newval>

When done splitting, start the balancer and undo the flags. Run upmap-remapped in a loop during PG splitting:

while true; do
    if ! ceph status 2>&1 | grep -q "peering\|activating\|laggy"; then
        ./upmap-remapped.py | sh -x
    fi
    sleep 30
done

This method allows the rebalancing to be deferred until after PG creation finishes, making the process more efficient.