Skip to main content

Bucket Sharding - increasing shard count

Context​

Bucket indexes are an essential data structure in RGW and are sharded over multiple RADOS objects to achieve more IOPS by parallelism.

The shard count of a bucket index comes with a set of trade-offs that must be considered as buckets grow. The conventional wisdom is that a single shard should not exceed 100,000 entries. If they do, Ceph issues a warning as performance might degrade. Unfortunately, more shards come with the problem that listing operations need to query more and more shards.

The process of resharding itself, up to and including Squid, is a write-blocking operation that takes time proportional to the number of objects in the bucket.

Resharding can be triggered manually or by dynamic resharding based on a shard size heuristic.

This document explains the resharding and dynamic resharding process in Ceph.

Background​

Bucket Indexes​

The bucket index is a RADOS object, or a set of objects, that tracks all the objects stored within a specific S3/Swift bucket. Every object uploaded to a bucket corresponds to a row in this index, recording metadata such as the object's name, size, ETag, and modification time.

The following code snippets show the most important bucket index entry fields that are OMAP values in a RADOS object:

struct rgw_bucket_dir_entry {
  cls_rgw_obj_key key;
  rgw_bucket_entry_ver ver;
  std::string locator;
  bool exists;
  rgw_bucket_dir_entry_meta meta; // see below
  std::multimap<std::string, rgw_bucket_pending_info> pending_map;
  uint64_t index_ver;
  std::string tag;
  uint16_t flags;
  uint64_t versioned_epoch;
}

struct rgw_bucket_dir_entry_meta {
  RGWObjCategory category = RGWObjCategory::None;
  uint64_t size = 0;
  ceph::real_time mtime;
  std::string etag;
  std::string owner;
  std::string owner_display_name;
  std::string content_type;
  uint64_t accounted_size = 0;
  std::string user_data;
  std::string storage_class;
  bool appendable = false;
  uint8_t restore_status = 0; // maps to RGWRestoreStatus enum
  ceph::real_time restore_expiry_date; // zero when N/A
}

The index is stored as RADOS OMAP, a key-value map inside a RADOS object. The index lives inside the OSD's storage engine.

To prevent a single RADOS object from becoming a bottleneck, it is sharded, and a hash function distributes objects across them. The shard count ceiling is 65,521. When a write arrives, the object's index entry is written to one specific shard determined by hashing the object name. When a Listing operation is called, RGW must query every shard and merge-sort the results into alphabetical order. If the number of shards is large relative to the object count, the bottleneck for listing becomes the number of shards rather than the number of objects.

Dynamic Resharding​

Dynamic resharding is an automatic process that monitors bucket index sizes and queues resharding.

There are 4 triggers that all need to be true for the dynamic resharding process to happen:

  1. rgw_dynamic_resharding == true
  2. Bucket layout is reshardable (not indexless, not mid-reshard)
  3. Current num_shards < rgw_max_dynamic_shards (default 1999)
  4. num_objs > num_shards × rgw_max_objs_per_shard (default 100,000)

When all four conditions are met, RGW adds an entry to the reshard queue. The entry records the bucket name, bucket ID, and the target shard count. The target shard count is calculated with the following formula:

target_shard_count = min(next_prime((num_objs × 2) / effective_max_objs_per_shard), rgw_max_dynamic_shards)

Each RGW process runs a ReshardWorker thread, which checks the reshard queue every 600 seconds and starts the resharding process on queued buckets.

Resharding (in Squid)​

Resharding is the process of changing the number of bucket index shards. The process involves copying all current bucket index entries. This is a blocking operation for write operations in Squid. GET operations are not blocked by the resharding process.

According to a Ceph blog post, resharding a 20-million-object bucket blocks writes for 4+ minutes, and a 500-million-object bucket requires 94 minutes of write unavailability.

Resharding can be triggered manually by an operator, or dynamic resharding can be enabled.

Resharding: Tentacle Improvements​

In Tentacle, bucket index resharding is now a two-phase process where the majority of the resharding occurs in the background without blocking I/O.

  1. First phase: the current index is copied to new shards while recording modifications into a journal.
  2. Second phase: we still briefly stop modifications in order to switch over to the new bucket index shards and replay the journal of concurrent modifications.

There is still a brief period where modifications stall. This is, however, much shorter than pre-Tentacle. According to the same blog post, the blocking period drops from several minutes to seconds in their test scenario.

Bucket Index Alternatives and Ongoing Work​

Indexless Buckets​

Indexless buckets skip the OMAP index entirely. On every PUT, RGW writes no index entry; the object lands in RADOS directly without any shard involvement. This eliminates resharding as a concern completely at the cost of disabling listing operations.

Ordered Bucket Indexes​

The current hash-based sharding distributes objects randomly across shards to balance load. The tradeoff is that listing always requires querying all shards and merging the results, with the cost scaling with shard count.

Ordered indexes instead maintain entries in lexicographic order within and across shards, so a listing scan can seek directly to the right position without touching shards that fall outside the requested prefix or range.

This approach also provides the option to reshard the individual shards instead of the whole bucket.

This feature is a work in progress in Tentacle and is planned for a future release.

Operational Guidance​

Pre-sharding on Bucket Creation​

The most reliable way to avoid unexpected resharding is to provision buckets with enough shards before they grow. Dynamic resharding only triggers after the threshold is already exceeded; pre-sharding prevents the threshold from being hit in the first place. The default shard count for new buckets can be set at the zone level via bucket_index_max_shards in the zonegroup config, or overridden via configuration (note: this requires an RGW daemon restart to take effect) using:

ceph config set rgw rgw_override_bucket_index_max_shards

Buckets can also be created with an explicit shard count via the S3 CreateBucket XML body or manually resharded at any time with:

radosgw-admin bucket reshard --bucket <name> --num-shards <new_shard_count>

It is best practice to pre-shard buckets during creation rather than relying on dynamic resharding. If the expected object count can be estimated, use it to calculate the initial number of shards.

Cleaning Up Temporary Entries​

When a LARGE_OMAP_OBJECTS alert fires, the excess key count may be due to temporary entries rather than actual object growth. The following entry types are cleanable without data loss:

  • Expired delete markers: versioned objects whose only remaining instance is a delete marker, removable via S3 Lifecycle ExpiredObjectDeleteMarker: true
  • Stale exists=false entries: orphaned index entries from interrupted operations, fixed by radosgw-admin bucket check --fix

Cleaning these up first can bring a shard below the OSD threshold without any resharding.