Skip to main content

Managing Ceph RGW Buckets with 160 Million Objects

Overview​

Managing a Ceph RADOS Gateway (RGW) bucket that contains 160 million objects presents significant operational challenges but is achievable with proper configuration and planning. This guide addresses key concerns and best practices for handling very large buckets in Ceph storage clusters.

Key Challenges​

1. Performance Considerations​

Bucket Index Performance

Ceph RGW bucket listing operations become noticeably slower with very large buckets. The primary bottleneck is the bucket index, which is stored in RADOS objects within the cluster.

Solution:

  • Ensure bucket index metadata is stored on SSDs
  • This can be accomplished using either:
    • HDD + SSD/NVMe hybrid OSD configuration (separate device classes)
    • All-flash OSD configuration
    • TLC flash storage is preferred over QLC flash storage
  • Use Ceph CRUSH rules to place bucket index pools on faster storage

2. Multisite Synchronization (RGW Multisite)​

Ceph RGW multisite replication becomes significantly more difficult at this scale. Syncing buckets across multiple Ceph clusters requires the RGW sync agents to process millions of objects.

Risk:

  • If synchronization breaks, resyncing the bucket can take many hours or even days
  • The larger the bucket, the longer recovery takes
  • RGW sync agents must iterate through all bucket index entries during recovery

Recommended Approach: Consider using multiple smaller buckets instead of a single 160-million object bucket. This provides several advantages:

  • Faster resync if sync breaks
  • Easier troubleshooting of sync issues
  • Better isolation of issues
  • More manageable recovery time objectives (RTOs)
  • Reduced load on individual RGW sync agents

Guidance on bucket size: To keep sync delays around 30 minutes, consider splitting objects across multiple buckets rather than maintaining a single massive bucket.

Note on Ceph versions:

  • Ceph Quincy and later have improved RGW multisite stability
  • Some advanced features (for example, enabling dynamic resharding) require Ceph Reef or later

3. rclone Synchronization with Ceph RGW​

Performance Reality: rclone performance when syncing between Ceph RGW endpoints is constrained by the time required to iterate through bucket objects, regardless of whether data needs to be transferred. rclone must list all objects via the S3 API to compare the source and the destination.

Real-World Example with Ceph:

  • A 50-million object bucket sync between two Ceph RGW clusters takes 1-2 days
  • This timing applies even when there are no objects to sync
  • Simply iterating over millions of objects through RGW's S3 interface takes many hours to days
  • Bucket index reads are the primary bottleneck

Implication for 160M objects: With 160 million objects (more than 3x the real-world example), expect proportionally longer sync times - potentially 3-6 days or more just to complete the iteration.

Why this happens with Ceph:

  • rclone uses S3 LIST operations to enumerate objects
  • Each LIST request to Ceph RGW queries the bucket index
  • Very large bucket indexes (even on SSDs) take time to traverse
  • Network latency between clusters adds overhead to each LIST operation

Recommendation: Consider the feasibility of using multiple buckets to improve sync performance and reduce operational complexity. Alternatively, consider using Ceph's native RGW multisite replication instead of rclone where possible.

4. RGW Bucket Index Sharding​

Sharding Requirements:

Ceph RGW uses bucket index sharding to distribute the bucket index across multiple RADOS objects. Each shard must contain fewer than 100,000 objects. .

Why sharding matters:

  • Bucket index is stored in RADOS objects
  • Without sharding, all index data would be in a single RADOS object
  • Large single objects cause performance degradation
  • Sharding distributes the index across multiple RADOS objects for parallel access

Calculation for 160M objects:

  • 160,000,000 objects ÷ 100,000 objects per shard = 1,600 minimum shards
  • Recommended: At least 1,500 shards

Current Configuration:

  • The default maximum number of shards is 1,999.
  • rgw_max_dynamic_shards can be used to configure the maximum number of shards. See the RADOSGW documentation
  • This exceeds the minimum requirement and provides good headroom
  • Configuration parameter: rgw_bucket_index_max_aio and bucket-specific shard settings

RGW Dynamic Resharding:

Dynamic resharding functionality in Ceph:

  • Very stable in recent Ceph versions (Quincy and later)
  • Automatically increases shard count as bucket grows
  • Important limitation: Cannot enable dynamic resharding with RGW multisite until Ceph Reef release
  • If using multisite, you must manually configure appropriate shard count before bucket growth

Manual sharding configuration:

# Set bucket shard count during creation
radosgw-admin bucket create --bucket=mybucket --num-shards=2003

# Or reshard an existing bucket (requires downtime or careful planning)
radosgw-admin bucket reshard --bucket=mybucket --num-shards=2003

Monitoring shard health:

# Check bucket stats including shard count
radosgw-admin bucket stats --bucket=mybucket

# List bucket index shards
radosgw-admin bi list --bucket=mybucket

Recommendations​

Short-term Actions​

  1. Get the CRUSH rule for the pool

    ceph osd pool get default.rgw.buckets.index crush_rule

  2. View the CRUSH rule to verify that it targets all-flash storage

    ceph osd crush rule dump <rule_name>

  3. Monitor sync performance - Establish baseline metrics for RGW multisite and rclone sync times

    # Check RGW sync status (if using multisite)
    radosgw-admin sync status

    # Monitor bucket sync lag
    radosgw-admin bucket sync status --bucket=<bucket-name>

Long-term Considerations​

  1. Evaluate bucket-splitting strategy

    • Consider the option to use multiple buckets
    • Target bucket sizes that keep sync delays under 30 minutes
    • Consider organizational or application boundaries for splitting objects
    • Use bucket-naming conventions to group related buckets logically

  2. Plan for Ceph Reef upgrade (if using RGW multisite)

    • Upgrade to a release that is still supported upstream. Reef ended end-of-life status in March 2026.
    • Dynamic resharding with multisite clusters didn't work prior to the Reef release (18.2.x).
    • This will provide more operational flexibility for growing buckets
    • Allows automatic shard count increases without manual intervention

  3. Optimize RGW configuration for large buckets

    # Recommended RGW settings for very large buckets
    ceph config set client.rgw rgw_bucket_index_max_aio 128
    ceph config set client.rgw rgw_thread_pool_size 512

  4. Disaster recovery planning

    • Document expected resync times (potentially days for 160M objects)
    • Establish procedures for monitoring and recovering from RGW sync breaks
    • Consider backup strategies beyond RGW multisite replication
    • Test bucket restoration procedures
    • Monitor RGW daemon health and sync agent performance

  5. Performance monitoring

    # Monitor RGW performance metrics
    ceph daemon /var/run/ceph/ceph-client.rgw.*.asok perf dump

    # Watch for bucket index bottlenecks
    ceph osd pool stats

General Considerations​

When considering this configuration:

  1. Set realistic expectations about sync times with buckets this large
  2. Propose multiple-bucket architecture as an alternative to improve:
    • Sync performance
    • Recovery times
    • Operational manageability
  3. Define acceptable sync delay SLAs (e.g., 30 minutes) and design bucket strategy accordingly
  4. Plan for growth - If the bucket will continue growing, establish thresholds for splitting

Summary​

A 160-million object bucket in Ceph RGW is feasible with proper configuration:

  • All-flash storage for bucket index (OSD configuration)
  • Adequate sharding (2003 shards via RGW bucket index sharding)
  • Be prepared for multi-day sync operations (RGW multisite or rclone)
  • Consider multiple-bucket strategy for better performance and manageability

The primary trade-off is operational complexity versus sync performance. Although a single large bucket is technically supported by Ceph RGW, multiple smaller buckets will provide:

  • Better RGW sync times
  • Easier recovery from sync failures
  • More manageable operations
  • Reduced load on bucket index
  • Improved RGW daemon performance

Ceph-specific considerations:

  • Bucket index stored as RADOS objects in .rgw.buckets.index pool
  • RGW performance depends on bucket index shard distribution
  • All-flash OSDs significantly improve bucket listing operations
  • RGW multisite is preferred over rclone for Ceph-to-Ceph replication
  • Monitor RGW daemon and sync agent health continuously