Definition

PG splitting with upmap-remapped is a technique used to speed up the PG- splitting process in Ceph clusters by preventing unnecessary data movement during splits.

How It Works

When pg_num is increased, PGs are split. When splitting a big data pool, the cluster must first resolve misplaced objects before making any further increase to pg_num.

This flow is as follows:

pgp_num gets increased -> creates misplaced objects -> wait for everything to rebalance -> pgp_num gets increased

The rebalancing of the PGs in a large data pool can take hours, and can sometimes takes days.

Use the upmap-remapped script (upmap-remapped.py) to defer balancing until after all the PGs have been split.

This flow is as follows:

pgp_num gets increased -> creates misplaced objects -> ./upmap-remapped removes misplaced objects -> pgp_num gets increased

Once this has finished and all new PGs have been created, the process of rebalancing begins.

# turn off balancing
ceph balancer off
ceph osd set norebalance

# increase the pg numbers
ceph osd pool set <pool> pg_num <newval>

When the splitting process has finished, start the balancer and undo the flags.

Run upmap-remapped in a loop during PG splitting:

while true; do
    if ! ceph status 2>&1 | grep -q "peering\|activating\|laggy"; then
        ./upmap-remapped.py | sh -x
    fi
    sleep 30
done

Monitor the cluster:

watch ceph -s
watch "ceph osd pool ls detail | grep data"

Monitor the Ceph internals dashboard.

Interrupt the loop to stop upmap-remapped runs by pressing Ctrl-C. After pg_num has reached the target value, interrupt the loop to stop upmap-remapped runs.

Turn on balancing:

ceph balancer on
ceph osd unset norebalance

This method allows rebalancing to be deferred until after PG creation has finished. This makes the rebalancing process more efficient.

This procedure was improved upon using input from Joel Davidow of NSO (jdavidow@nso.edu).