S3 Usage Guidelines
Introduction
In principle, S3 object storage imposes no hard limits on bucket size, object count, object size, or request rate. In practice, however, certain usage patterns can significantly affect performance and stability. The following guidelines are intended to help users get the best out of the service and to avoid patterns that have caused serious problems in the past.
Number of Objects per Bucket
As a bucket grows larger, some operations begin to slow down. Listing the contents of a bucket (LIST operations) is particularly affected. Accessing or retrieving individual objects does not degrade with bucket size, but creating new objects in a very large bucket will slow down.
Users should aim to keep the number of objects per bucket below 10–50 million whenever possible.
It has also been demonstrated repeatedly that a single bucket can become a performance bottleneck. Users who have performance- or latency-sensitive workloads are strongly advised to shard their data across multiple buckets from the outset, rather than attempting to reorganise later.
Automated bucket sharding helps to split the index as the bucket grows in size. This ensures scalability.
Object Size
Large Objects
Large objects do not pose any particular issue and can be stored without concern.
Small Objects
Small objects are a significant concern, particularly in deployments that use erasure coding (EC). Objects smaller than the size of a full stripe will incur write amplification. For example, in a 4+2 erasure coding configuration, a full stripe is six times the minimum allocation unit (approximately 32kB), meaning that very small objects consume disproportionately more storage and I/O than their size would suggest.
Small objects in an EC pool have caused serious operational problems in the past, and are considered a contributing factor in significant outages. Users who expect to store large numbers of small objects should discuss their use case with the storage team before proceeding.
Routing Objects by Size
It is possible to configure the system so that objects below a threshold size are stored on replicated pools, while larger objects are stored on an EC pool. This can be achieved using RGW placement targets and LocationConstraints in PUT requests. Users who require this kind of routing should contact the storage team to discuss their requirements.
LocationConstraints can be used to set the default placement pool of a bucket
at creation so that all objects written to the bucket will, by default, end
up in a replicated or EC pool.
Routing objects based on their size can be achieved by the client choosing the appropriate storage class during upload, or — when the client does not do so — by RGW admins using RGW Lua scripting and custom rules to set or override the storage class on upload. This enables fine-grained, optimized data placement that is consistently enforced, even when clients don't cooperate.
See also the Storage Classes section of the RGW documentation.
See also the Lua-scripting page of the Ceph documentation for more on routing objects to different storage classes.
See also Frederic Nass's S3 Dynamic Placement and Archiving repository for more on advanced data-placement scenarios.
Object Count and PG Sizing
The number of objects stored per OSD and per placement group (PG) has a significant effect on cluster health and performance. As a worked example: with a 4+3 EC configuration and an average object size of around 1MB, each object occupies approximately 256kB per OSD across 7 OSDs. A 15TB OSD would therefore hold around 60 million objects. With the recommended 100 PGs per OSD, that amounts to around 600,000 objects per PG.
Although this is technically manageable, it is near the high end of what is advisable. In the field, PGs holding more than 1M objects have been observed to develop problems related to database performance, backfilling between OSDs, and scrubbing. The ideal range is 100,000–200,000 objects per PG, with up to one million considered manageable.
Minimum Allocation Size and Billing
BlueStore's minimum allocation size (min_alloc_size) determines the smallest
unit of storage that can be written. Objects smaller than this threshold are
padded with zeros to fill the minimum allocation unit. The relevant thresholds
are:
- Replicated pools: 4kB minimum allocation size. Objects larger than 4kB incur no padding overhead.
- EC 4+3 pools: 16kB minimum allocation size. Objects smaller than 16kB will be padded with zeros.
While padding overheads do not typically add up to a very large amount of wasted storage in absolute terms (for example, one billion 16kB objects wasted amounts to 16TB total), they are a real cost and should be factored into capacity planning.
For billing purposes, it is reasonable to charge according to the minimum allocation unit — 4kB for replicated pools and 16kB for EC pools — since this reflects the actual storage consumed regardless of the object's logical size.
Request Rates
There is no single acceptable request rate that applies to all workloads. The appropriate rate depends heavily on the type of requests being made. High-throughput bulk data transfers and high-frequency small requests for small files (such as web assets) have very different characteristics and can interfere with each other if not properly separated.
Users with demanding or latency-sensitive workloads should discuss their requirements with the storage team so that appropriate arrangements can be made.
RGW usage and ops logs can have an adverse effect on performance if not configured optimally. Our recommendations for the ops log are documented at the S3 Operations Logging page.
Summary
| Consideration | Guideline |
|---|---|
| Objects per bucket | Aim to stay below 10–50 million |
| Multiple buckets | Recommended for performance-sensitive workloads |
| Large objects | No known issues |
| Small objects (EC pools) | Avoid where possible; discuss with storage team |
| Objects per PG | Ideal 100k–200k; up to 1 million manageable |
| Min allocation size (replicated) | 4kB |
| Min allocation size (EC 4+3) | 16kB |
| Minimum billable unit | Equal to minimum allocation size |
| Request rates | Depends on workload type; contact storage team for guidance |