9 posts tagged with "radosgw"

Below are a list of known bugs in ceph versions that we want to highlight. As of time of writing this, the latest versions for each release are the following:

• reef: 18.2.7
• squid: 19.2.3

There are more bugs in Ceph that were not included here, we've highlighted a few that we wanted to share.

Cross-version Issues​

These critical bugs affect multiple major versions of Ceph.

RadosGW --bypass-gc Data Loss Bug​

Severity: Critical
Affected Versions: Quincy (17.2.x), Reef (18.2.x), Squid (19.2.x) Bug Tracker: https://tracker.ceph.com/issues/73348

Description​

An RGW bug that has affected all ceph versions. With --bypass-gc causes deletion of copied object data.

When using radosgw-admin bucket rm --bypass-gc, if any of the deleted objects had been copied to/from other buckets using the server-side S3 CopyObject operation, the command deletes the data of those copies as well. As a result, the copied objects remain visible in ListObjects requests but GetObject requests fail with NoSuchKey errors, leading to silent data loss.

**This bug affects all major versions: Quincy (v17), Reef (v18), Squid (v19) Backports are still in progress for reef, squid and tentacle releases.

Recommendation​

• Do not use --bypass-gc flag - Use radosgw-admin bucket rm without --bypass-gc which correctly handles copied objects
• If you've used --bypass-gc in the past, audit your buckets for objects with missing data
• Follow the bug tracker at https://tracker.ceph.com/issues/73348 for fix and backport updates

Ceph Squid (v19) Known Issues​

Squid Deployed OSDs Are Crashing​

Severity: Critical
Affected Versions: 19.2.*
Bug Tracker: https://tracker.ceph.com/issues/70390

Description​

OSDs created in v19 may crash unexpectedly. This issue affects only newly deployed OSDs using Squid, while previously deployed OSDs continue to run fine. The root cause is likely the Elastic Shared Blob implementation introduced in PR #53178. Unfortunately, ceph-bluestore-tool repair cannot fix affected OSDs.

Recommendation​

• Before deploying new OSDs: Run ceph config set osd bluestore_elastic_shared_blobs 0 on any Squid cluster
• For already affected OSDs: The only known fix is complete redeployment of the OSD

S3 DeleteBucketLifecycle Does Not Delete Config​

Severity: Medium
Affected Versions: 19.2.3
Bug Tracker: https://tracker.ceph.com/issues/71083

Description​

The S3 DeleteBucketLifecycle API call fails to actually delete the lifecycle configuration from the bucket, causing S3 compatibility issues. This is a regression introduced specifically in squid 19.2.3.

Recommendation​

• Upgrade to a fixed version when available
• Avoid using the DeleteBucketLifecycle API in affected versions
• Consider using alternative methods to manage lifecycle configurations

Ceph Reef (v18) Known Issues​

BlueStore Potential Corruption​

Severity: Critical
Affected Versions: 18.2.5, 18.2.6
Bug Tracker: https://tracker.ceph.com/issues/69764

Description​

Versions 18.2.5 and 18.2.6 of Ceph Reef were released with a bug that may cause OSDs to crash and corrupt on-disk data. This is a data integrity issue that poses serious risks to cluster stability and data safety.

Recommendation​

Upgrade to version 18.2.7. Do not deploy or upgrade to versions 18.2.5 or 18.2.6.

Ceph Quincy (v17) Known Issues​

BlueStore Potential Corruption​

Severity: Critical
Affected Versions: 17.2.8
Bug Tracker: https://tracker.ceph.com/issues/69764

Description​

Version 17.2.8 of Ceph Quincy was released with a bug that may cause OSDs to crash and corrupt on-disk data. This is a data integrity issue that poses serious risks to cluster stability and data safety.

Recommendation​

Upgrade to version 17.2.9 or 18.2.7. Do not deploy or remain on version 17.2.8.

Additional Resources​

Please file a ticket in our support portal if there are any other questions or concerns.

As of now, we recommend Reef v18.2.7 as a stable version to be used in production. This changes overtime as we gain more confidence over later ceph releases.

For more detailed information and updates on these issues, refer to our knowledge base articles:

• Ceph Quincy (v17) Known Bugs
• Ceph Reef (v18) Known Bugs
• Ceph Squid (v19) Known Bugs

Commvault has been in use as a data protection solution for years and is now looking to replace its existing storage solution (EMC), for its entire customer environments.

Commvault provides data backup through a single interface. Through the gradual deployment of Ceph S3 in several expansion stages, the customer built confidence in Ceph as a storage technology and more and more backups are gradually being transferred to the new backend.

In the first phase, Ceph S3 was allowed to excel in its performance and scalability capabilities.

In the following phases, the focus will be on flexibility and use as unified storage for cloud computing and Kubernetes.

For all these scenarios, the customer relies on Ceph as an extremely scalable, high-performance and cost-effective storage backend.

Over 1 PB of backup data and more than 500 GBytes per hour of backup throughput can be easily handled by Ceph S3 and it is ready to grow even further with the requirements in the future.

After in-depth consultation, we were able to exceed the customer’s expectations for the Ceph cluster in production.

The customer uses Commvault as a data backup solution for their entire customer environments.

Wherever the data resides, Commvault provides the backup of the data through a single interface. The customer thus avoids costly data loss scenarios, disconnected data silos, lack of recovery SLAs and inefficient scaling.

For all these scenarios, the customer relies on Ceph as a powerful and cost-effective storage backend for Commvault.

With over 2 PB of backup data and more than 1 TByte per hour of backup throughput, Ceph can easily handle and is ready to grow even further with the requirements in the future.

In conclusion, we were able to clearly exceed the customer’s expectations of the Ceph Cluster already in the test phase.

motivation​

we have tested ceph s3 in openstack swift intensively before. We were interested in the behavior of the radosgw stack in ceph. We paid particular attention to the size and number of objects in relation to the resource consumption of the radosgw process. Effects on response latencies of radosgw were also important to us. To be able to plan the right sizing of the physical and virtual environments.

technical topics​​

From a technical point of view, we were interested in the behavior of radosgw in the following topics.

• dynamic bucket sharding
• http frontend difference between Civetweb and Beast
• index pool io pattern and latencies
• data pool io pattern and latencies with erasure-coded and replicated pools
• fast_read vs. standard read for workloads with large and small objects.

requirements​

when choosing the right tool, it was important for us to be able to test both small and large ceph clusters with several thousand osds.

We want to use the test results as a file for evaluation as well as have a graphical representation as timeseries data.

For timeseries data we rely on the standard stack with Grafana, Prometheus and Thanos.

the main prometheus exporters we use are ceph-mgr-exporter and node-exporter.

load and performance tools​​

CBT - The Ceph Benchmarking Tool​

CBT is a testing harness written in python

https://github.com/ceph/cbt

s3 - tests​

This is a set of unofficial Amazon AWS S3 compatibility tests

https://github.com/ceph/s3-tests

COSBench - Cloud Object Storage Benchmark​

COSBench is a benchmarking tool to measure the performance of Cloud Object Storage services.

https://github.com/intel-cloud/cosbench

Gosbench​

Gosbench is the Golang reimplementation of Cosbench. It is a distributed S3 performance benchmark tool with Prometheus exporter leveraging the official Golang AWS SDK

https://github.com/mulbc/gosbench

hsbench​

hsbench is an S3 compatable benchmark originally based on wasabi-tech/s3-benchmark.

https://github.com/markhpc/hsbench

Warp​

Minio - S3 benchmarking tool.

https://github.com/minio/warp

the tool of our choice​​

getput​

getput can be run individually on a test client.

gpsuite is responsible for synchronization and scaling across any number of test clients. Communication takes place via ssh keys and the simultaneous start of all s3 test clients is synchronized over a common time base.

Installation on linux as script or as container is supported.

https://github.com/markseger/getput

radosgw-admin key create --uid=clyso-user-id --key-type=s3 --gen-access-key --gen-secret

...

"keys": [
{
"user": "clyso-user-id",
"access_key": "VO8C17LBI9Y39FSODOU5",
"secret_key": "zExCLO1bLQJXoY451ZiKpeoePLSQ1khOJG4CcT3N"
}
],

...

access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/object_gateway_guide_for_red_hat_enterprise_linux/administration_cli#create_a_key

List of users:

radosgw-admin metadata list user

List of buckets:

radosgw-admin metadata list bucket

List of bucket instances:

radosgw-admin metadata list user.instance

All necessary information

• user-id = Output from the list of users
• bucket-id = Output from the list of bucket instances
• bucket-name = Output from the list of buckets or bucket instances
• Change of user for this bucket instance:

radosgw-admin bucket link --bucket <bucket-name> --bucket-id <default-uuid>.267207.1 --uid=<user-uid>

Example:

radosgw-admin bucket link --bucket test-clyso-test --bucket-id aa81cf7e-38c5-4200-b26b-86e900207813.267207.1 --uid=c19f62adbc7149ad9d19-8acda2dcf3c0

If you compare the buckets before and after the change, the following values are changed:

• ver: is increased
• mtime: will be updated
• owner: is set to the new uid
• key: user.rgw.acl: The rights are reset for the user.rgw.acl key

Example to remove authorizations of a user with radosgw-admin

Remove the authorization of buckets=read from the user clyso-user-id.

radosgw-admin caps rm --uid=clyso-user-id --caps="buckets=read"

The aim is to achieve a scaling of the rgw instances for the production system so that 10,000 active connections are possible.

As a result of various test runs, the following configuration emerged for our setup

[client.rgw.<id>]
keyring = /etc/ceph/ceph.client.rgw.keyring
rgw content length compat = true
rgw dns name = <rgw.hostname.clyso.com>
rgw enable ops log = false
rgw enable usage log = false
rgw frontends = civetweb port=80
error_log_file=/var/log/radosgw/civetweb.error.log
rgw num rados handles = 8
rgw swift url = http://<rgw.hostname.clyso.com>
rgw thread pool size = 512

Notes on the configuration​

rgw thread pool size ist der Standardwert für num_threads des civeweb webservers.

Line 54: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_civetweb_frontend.cc

set_conf_default(conf_map, "num_threads",
std::to_string(g_conf->rgw_thread_pool_size));
[client.radosgw]
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw content length compat = true
rgw dns name = <fqdn hostname>
rgw enable ops log = false
rgw enable usage log = false
rgw frontends = civetweb port=8080 num_threads=512
error_log_file=/var/log/radosgw/civetweb.error.log
rgw num rados handles = 8
rgw swift url = http://<fqdn hostname>
rgw thread pool size = 51``

sources​

https://github.com/ceph/ceph/blob/master/doc/radosgw/config-ref.rst

http://docs.ceph.com/docs/master/radosgw/config-ref/

https://github.com/ceph/ceph/blob/master/src/rgw/rgw_civetweb_frontend.cc

https://indico.cern.ch/event/578974/contributions/2695212/attachments/1521538/2377177/Ceph_pre-gdb_2017.pdf

http://www.osris.org/performance/rgw.html

https://www.swiftstack.com/docs/integration/python-swiftclient.html

https://github.com/civetweb/civetweb/tree/master/docs

If you quickly need the syntax for the radosgw-admin command.

clyso-ceph-rgw-client:~/clyso # radosgw-admin object stat --bucket=size-container --object=clysofile


{
"name": "clysofile",
"size": 26,
"policy": {
"acl": {
"acl_user_map": [
{
"user": "clyso-user",
"acl": 15
}
],
"acl_group_map": [],
"grant_map": [
{
"id": "clyso-user",
"grant": {
"type": {
"type": 0
},
"id": "clyso-user",
"email": "",
"permission": {
"flags": 15
},
"name": "clyso-admin",
"group": 0,
"url_spec": ""
}
}
]
},
"owner": {
"id": "clyso-user",
"display_name": "clyso-admin"
}
},
"etag": "clyso-user",
"tag": "d667b6f1-5737-4f5e-bad0-fc030f0a4e94.11729649.143382",
"manifest": {
"objs": [],
"obj_size": 26,
"explicit_objs": "false",
"head_size": 26,
"max_head_size": 4194304,
"prefix": ".ZQzVc6phBAMCv3lSbiHBo0fftkpXmjm_",
"rules": [
{
"key": 0,
"val": {
"start_part_num": 0,
"start_ofs": 4194304,
"part_size": 0,
"stripe_max_size": 4194304,
"override_prefix": ""
}
}
],
"tail_instance": "",
"tail_placement": {
"bucket": {
"name": "size-container",
"marker": "d667b6f1-5737-4f5e-bad0-fc030f0a4e94.11750341.561",
"bucket_id": "d667b6f1-5737-4f5e-bad0-fc030f0a4e94.11750341.561",
"tenant": "",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
}
},
"placement_rule": "default-placement"
}
},
"attrs": {
"user.rgw.pg_ver": "��",
"user.rgw.source_zone": "eR[�\u0011",
"user.rgw.tail_tag": "d667b6f1-5737-4f5e-bad0-fc030f0a4e94.11729649.143382",
"user.rgw.x-amz-meta-mtime": "1535100720.157102"
}
}