Skip to main content

When and How to Enable Client Auto Eviction

Problem​

In some customer environments, an unknown workload is regularly causing MDS_CLIENT_RECALL, MDS_SLOW_REQUEST, MDS_CLIENT_OLDEST_TID warnings. Operators are using the client "eviction" procedure as a workaround and would like some automation.

Solution​

Automatic client eviction should be used only sparingly, after the following conditions have been satisfied:

  1. The CephFS cluster is seeing MDS_CLIENT_RECALL warnings lasting many hours, with MDS_SLOW_REQUEST ops also lasting many hours.
  2. Manual client eviction is confirmed to resolve the MDS_SLOW_REQUEST warnings fully.
  3. Manual client eviction is confirmed with the client/user to not have an adverse impact on their workload or data consistency.
  4. The workload does not write the same files from more than one client, or it tolerates a write from an evicted client overwriting newer data (see Data safety below).

If all of the above are true, then you may configure automatic client eviction, e.g. after 15 minutes of blocked caps eviction. Run the following commands to configure automatic client eviction:

# ceph config set mds mds_session_blocklist_on_evict false
# ceph config set mds mds_session_blocklist_on_timeout false
# ceph config set mds mds_cap_revoke_eviction_timeout 900

Data safety​

With mds_session_blocklist_on_evict and mds_session_blocklist_on_timeout set to false, an evicted client is not fenced from the OSDs. Eviction ends its MDS session, but the client can still write to the OSDs, so data it had buffered when it was evicted can reach the OSDs later, after the MDS has given the file to another client. That late write can overwrite data the other client wrote in the meantime.

We reproduced this with libcephfs and ceph-fuse clients: a client with unflushed writes is evicted, a second client writes the same range of the file and flushes, and the first client's stale data then lands on top. It happened in every run with blocklisting disabled, whether or not the client sets client_reconnect_stale.

Blocklisting (the Ceph default) is what prevents this: the OSDs reject the evicted client's writes. Note that a libcephfs or ceph-fuse client with client_reconnect_stale = true reconnects under a new address after being blocklisted and can then flush data it buffered before the eviction, which defeats the blocklist; a fix is under review upstream in ceph/ceph#73525. Until that fix is in your release, leave client_reconnect_stale at its default (false) where clients share written files.

Disable blocklisting only for workloads where each file is written from a single client, or where such an overwrite is acceptable.