validate if the RBD Cache is active on your client
By default the cache is enabled, since Version 0.87.
To enable the cache on the client side you have to add following config /etc/ceph/ceph.conf:
[client]
rbd cache = true
rbd cache writethrough until flush = true
add local admin socket
So that you can also verify the status on the client side, you must add the following two parameters:
[client]
admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/ceph/
configure permissions and security
Both paths must be writable by the user who uses the RBD library. Applications such as SELinux or AppArmor must be properly configured.
request infos via admin socket
Once this is done, run your application that is supposed to use librbd (kvm, docker, podman, ...) and request the information via the admin daemon socket:
$ sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.66606.140190886662256.asok config show | grep rbd_cache "rbd_cache": "true", "rbd_cache_writethrough_until_flush": "true", "rbd_cache_size": "33554432", "rbd_cache_max_dirty": "25165824", "rbd_cache_target_dirty": "16777216", "rbd_cache_max_dirty_age": "1", "rbd_cache_max_dirty_object": "0", "rbd_cache_block_writes_upfront": "false",
Verify the cache behaviour
To compare the performance difference you can test the cache, you can deactivate it in the [client] section in your ceph.conf as follows:
[client]
rbd cache = false
Then run a fio benchmark with the following command:
fio --ioengine=rbd --pool=<pool-name> --rbdname=rbd1 --direct=1 --fsync=1 --rw=write --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_base
Finally, run this test with RBD client cache enabled and disabled and you should notice a significant difference.
Sources
https://www.sebastien-han.fr/blog/2015/09/02/ceph-validate-that-the-rbd-cache-is-active/