Skip to main content

Resolving MDS_HEALTH_TRIM Warnings

Problem​

The cluster is experiencing MDS_HEALTH_TRIM warnings, indicating that the MDS journal is not being trimmed efficiently and may be growing beyond expected thresholds.

Solution​

The approach depends on whether the MDS_HEALTH_TRIM warnings are intermittent or persistent:

For Intermittent Warnings (Bursty Workloads)​

If the warnings are intermittent, it typically indicates that client workload is bursty and the MDS cannot keep up with those bursts. To silence these warnings:

# ceph config set mds mds_log_warn_factor 4

You can also try a value of 5 if needed. This increases the threshold before warnings are triggered.

For Persistent Warnings (Journal Trimming Issues)​

  1. Increase the maximum journal segments to allow the MDS to handle more segments before trimming:
# ceph config set mds mds_log_max_segments 256

Note: Larger values will increase MDS failover time as more journal needs to be replayed.

  1. Disable load balancing if running multiple MDS without subtree pinning:
# ceph config set mds mds_bal_interval 0

This prevents workload exchanges between MDSs which can interfere with journal trimming.

Discussion​

Relevant Configuration Options​

  • mds_log_events_per_segment (default: 1024): Maximum number of events in an MDS journal segment. Do not modify this value.

  • mds_log_max_segments (default: 128): Maximum number of segments which may be untrimmed.

  • mds_log_warn_factor (default: 2): Triggers MDS_HEALTH_TRIM warning when the MDS log is longer than mds_log_max_segments × mds_log_warn_factor.

  • mds_bal_interval (default: 10): The frequency (in seconds) of workload exchanges between MDSs.

  • mds_cap_revoke_eviction_timeout (default: 0): Number of seconds after which clients that have not responded to cap revoke messages are evicted.

Root Cause Analysis​

MDS_HEALTH_TRIM warnings can occur when:

  • Client workloads are bursty and overwhelm the MDS trimming capability
  • Multiple MDS instances without proper subtree pinning create load balancing overhead
  • The MDS cannot trim journal segments fast enough to keep up with incoming operations