Skip to main content

Slow Small-Object GETs from RGW: Enable tcp_nodelay

Symptoms​

  • S3 GETs of small objects (up to a few tens of KiB) take about 40 ms each, even on an idle cluster with fast OSDs.
  • Each keep-alive connection serves only about 25 small GETs per second, so clients with a fixed number of connections see very low throughput.
  • Larger objects are faster than small ones: a 64 KiB GET completes in a millisecond or two, while a 4 KiB GET takes 40 ms.
  • RGW and OSD CPU stay low while clients wait.

Cause​

RGW's beast frontend enables the TCP_NODELAY socket option only when rgw_frontends contains tcp_nodelay=1. The default is 0, which leaves Nagle's algorithm enabled. Neither cephadm nor Rook sets the option.

RGW sends the response headers and the response body as two separate writes. With Nagle's algorithm enabled, the second write is held back while the first is unacknowledged, if it is smaller than one TCP segment. That is the case for a small body. The client delays its acknowledgement of the headers by up to 40 ms on Linux (delayed ACK), so every small response waits for that timer.

This affects all Ceph releases with the beast frontend, including Clyso Enterprise Storage releases up to and including the 20.2 series. Bodies larger than a TCP segment are not delayed.

How to Check​

Fetch the same small object twice over one connection. curl reuses the connection when given the URL twice; the second request shows the delay:

curl -s -o /dev/null -o /dev/null -w '%{time_total}\n' \
  http://<rgw-host>:<port>/<bucket>/<small-object> \
  http://<rgw-host>:<port>/<bucket>/<small-object>

The first request on a new connection is not delayed. If the second time is about 0.04 s, Nagle's algorithm is delaying the response:

0.003520
0.041196

With tcp_nodelay=1, both are around a millisecond. Check the frontend configuration of the RGW daemons:

ceph config dump | grep rgw_frontends

If no rgw_frontends value contains tcp_nodelay=1, the option is off.

Resolution​

cephadm​

Add tcp_nodelay=1 to rgw_frontend_extra_args in the RGW service specification and apply it:

service_type: rgw
service_id: <service-id>
placement:
  ...
spec:
  rgw_frontend_port: 8080
  rgw_frontend_extra_args:
  - "tcp_nodelay=1"
ceph orch apply -i rgw-spec.yaml

cephadm updates rgw_frontends for each daemon and redeploys the RGW daemons. Keep any extra arguments the specification already has.

Manually Managed RGW Daemons​

Append tcp_nodelay=1 to the existing rgw_frontends value of each RGW daemon, then restart it:

ceph config get client.rgw.<name> rgw_frontends
ceph config set client.rgw.<name> rgw_frontends "beast port=8080 tcp_nodelay=1"
systemctl restart ceph-radosgw@rgw.<name>

Keep the other options of the existing value, such as ssl_port and ssl_certificate.

Rook​

Rook builds the rgw frontends value itself and passes it on the command line, so a rgw_frontends value in the configuration database has no effect. Contact Clyso support for the options available for your Rook version.

Results​

Measured on a test cluster with one RGW and 16 clients that each keep one connection open:

Workloadtcp_nodelay=0 (default)tcp_nodelay=1
4 KiB GET, requests per second3819,039
4 KiB GET, median latency42 ms1.7 ms
64 KiB GET, requests per second8,2198,639

Disabling Nagle's algorithm does not increase RGW CPU use per request. It can increase the number of small packets on the network, which matters little for S3 traffic.

  • The upstream Ceph change that makes tcp_nodelay=1 the default (ceph/ceph#73475, tracker #82657), and a Clyso Enterprise Storage update with the same default, are in progress. Until they are released, set the option as described above.
  • rgw_thread_pool_size also affects RGW CPU use per request. Clyso Enterprise Storage sets it to 128 instead of the upstream Tentacle default of 512.