Skip to main content

RGW Localized Reads on Ceph Tentacle

Problem​

RGW reads from a replicated pool default to the PG's primary OSD, regardless of where that OSD sits relative to the RGW serving the request. In a cluster spread across datacenters, this means a lot of reads cross racks even when a local replica is sitting right there. That adds latency and consumes bandwidth between racks for no reason.

On a test cluster with three racks and size=3 (Ceph 20.2.4 / Tentacle), with no locality setting applied, only 29.2% of read bytes came from the local OSD. That matches almost exactly how often that OSD happened to be primary (14/48 objects, 29.17%). Reads just follow the primary. Locality is otherwise down to luck.

Implementation​

Two settings, both required, on the RGW client side:

ceph config set client.rgw.<service>.<host>.<suffix> crush_location "root=default datacenter=<datacenter> rack=<rack> host=<host>"
ceph config set client.rgw.<service>.<host>.<suffix> rados_replica_read_policy localize
ceph orch restart rgw.<service>
  • crush_location tells the RGW its position in the CRUSH tree. Alone, it does nothing. This was tested and confirmed to be identical, byte for byte, to the baseline with no setting applied. Target the specific daemon entity, not the generic client.rgw section, or every RGW believes it's in the same place.
  • rados_replica_read_policy=localize makes the Objecter compute, per read, the closest OSD in the PG's acting set to the client's declared location (the nearest common CRUSH bucket type wins; primary wins ties).
  • A restart is mandatory. ceph config help reports the policy as updatable at runtime. That's false in practice. The Objecter caches it at startup; ceph config set while running shows up on the admin socket but changes nothing until the daemon restarts. (ceph orch daemon restart refuses to restart the last RGW in a service. It needs --force or a second instance.)

Scope:

  • Replicated pools only. EC has to reconstruct from shards, so no single replica can answer alone.
  • Locality is total only when the number of racks is greater than or equal to the replication size, forcing every acting set to include the local rack. When racks are fewer than replicas, only some PGs get a local hit, so locality becomes partial.
  • Bucket index and other RADOS class call reads (rgw.bucket_list, rgw.gc_list, etc.) localize automatically as soon as localize is set. The separate rados_replica_read_policy_on_objclass flag showed no measurable effect either way.
  • There is no availability risk. If the local OSD goes down, reads fall back to the primary with no errors.

Verification​

Use op_r_out_bytes rather than raw op_r. Background traffic adds ops but negligible bytes, so it's the cleaner signal.

  1. Confirm the local OSD is actually in the acting set for the PGs you're testing (ceph pg dump pgs_brief, ceph osd tree). Nothing works if it isn't.
  2. Pin pg_num (autoscaler off) and set noscrub/nodeep-scrub for the test window so the mapping and background I/O don't move underneath you.
  3. Baseline: ceph tell osd.<id> perf dump | grep -E '"op_r"|"op_r_out_bytes"' on each OSD before generating load.
  4. Generate a known read volume (a fixed set of GETs, known total size), then sample the same counters again.
  5. Apply the two settings, restart, and repeat the load. With the topology condition met, the local OSD's op_r_out_bytes delta should account for essentially 100% of bytes served (reference test: 1,006,743,283 of 1,006,743,283 bytes; other OSDs showed a few residual ops with zero bytes). If bytes are still spread proportional to primary assignment, check three things: was the RGW actually restarted, was crush_location set on the right entity, and does the local rack genuinely appear in the tested PGs' acting sets.
  6. To confirm the restart requirement specifically, flip the policy back to default with no restart and verify behavior is unchanged (still fully local). Only after restarting should it fall back to baseline.