Skip to main content

Increasing Shard Count

Problem​

Many small objects are created, and it is necessary to increase the shard count for almost every bucket. Error large_omap_object is thrown.

The question is this: If many objects are uploaded to a new bucket and we know that the bucket will hit the threshold where it automatically reshards, is it advisable to manually reshard the bucket, or does manually resharding cost more than it saves?

Solution​

Start with a higher shard count if you are regularly observing resharding events. Every time buckets reshard, they block IO during the resharding process and this can get expensive for very large buckets. The biggest evident downside to having a high initial bucket shard count is that bucket listing performance degrades as the shard count grows. For example, for buckets that don't need lots of shards, you'll have worse listing performance than if you used a lower default shard count.

You can override a zonegroup's default setting by running a command of the following form:

radosgw-admin zonegroup modify --bucket-index-max-shards=<shards>

or by setting rgw_override_bucket_index_max_shards in the global configuration to the new default shard count if you want to change the setting universally. Use a prime number if you decide to change the default shard count. All RGW daemons in the cluster must be restarted in order for the new setting to take effect.

Discussion​

Here are the advantages of having more initial shards:

  • Better initial PUT/GET throughput
  • Fewer objects per shard
  • Initial resharding is avoided

Here is a disadvantage of having more initial shards:

  • Initial listing performance is worse

See the following pull request for more of the history around this topic: https://github.com/ceph/ceph/pull/30875