Ceph cluster health state​

root@clyso-ceph:/home/ceph# ceph -s
  cluster:
    id:     FEBB01CC-4AA5-4C1D-80C4-9D91901467C8
    health: HEALTH_WARN
            256 pgs not deep-scrubbed in time
            266 pgs not scrubbed in time

Admin has noticed that deep-scrubbing has fallen behind.

Possible actions​

If impacting ceph performance is not a concern. osd max scrubs and osd scrub load threshold can be carefully be adapted, but that can have a huge impact on ceph cluster performance.

Show current config​

root@clyso-ceph:/home/ceph# ceph config show osd.1 osd_max_scrubs
1
root@clyso-ceph:/home/ceph#

root@clyso-ceph:/home/ceph# ceph config get osd osd_scrub_load_threshold
0.500000
root@clyso-ceph:/home/ceph#

Set osd_max_scrub​

ceph config set osd_max_scrubs 2

Verify setting​

Ceph config database​

root@ceph-vm-az1-1:/home/kraftmayerj# ceph config get osd osd_max_scrubs
2
root@ceph-vm-az1-1:/home/kraftmayerj#

Ceph osd active settings (osd.1)​

root@clyso-ceph:/home/ceph# ceph config show osd.1 osd_max_scrubs
2
root@clyso-ceph:/home/ceph#

Sources​

osd_max_scrubs

osd_scrub_load_threshold

osd_scrub_during_recovery

On the topic of "strong consistency" of rados, there is a reference in the upstream document [1] to Sage's original paper on rados and a chapter "3.2 Strong Consistency" there.

[1] https://docs.ceph.com/en/reef/architecture/#the-ceph-storage-cluster

[2] https://ceph.io/assets/pdfs/weil-rados-pdsw07.pdf

Hello Ceph community! It's time again for another blog post! One of the most common questions we've gotten over the years is whether or not users should deploy multiple OSDs per flash drive. This topic is especially complicated because our advice has changed over the years. Back in the Ceph Nautilus era, we often recommended 2, or even 4 OSDs per flash drive. There were obvious and significant performance advantages at the time when deploying multiple OSDs per flash device, especially when using NVMe drives.

We habe backuped kubernetes cluster before using Ark.

The Ark project was then acquired by VMWare circa 2019 and renamed Velero.

Currently, we are supporting another customer in backing up data stored in multiple Kubernetes environments to be ready for disaster recovery.

Hello Ceph community! Here at Clyso we’ve been thinking quite a bit about the tuning defaults and hardware/software recommendations we will be making for users of our upcoming Clyso Enterprise Storage (CES) product based on Ceph. We decided that given how useful some of this information is both for CES and for the upstream project, we’d open the document up to the community for feedback and to help us build a better product. We’ll be adding more content as time goes on. Feel free to reach out at mark.nelson at clyso.com if you have thoughts or questions!

Download

Commvault has been in use as a data protection solution for years and is now looking to replace its existing storage solution (EMC), for its entire customer environments.

Commvault provides data backup through a single interface. Through the gradual deployment of Ceph S3 in several expansion stages, the customer built confidence in Ceph as a storage technology and more and more backups are gradually being transferred to the new backend.

In the first phase, Ceph S3 was allowed to excel in its performance and scalability capabilities.

In the following phases, the focus will be on flexibility and use as unified storage for cloud computing and Kubernetes.

For all these scenarios, the customer relies on Ceph as an extremely scalable, high-performance and cost-effective storage backend.

Over 1 PB of backup data and more than 500 GBytes per hour of backup throughput can be easily handled by Ceph S3 and it is ready to grow even further with the requirements in the future.

After in-depth consultation, we were able to exceed the customer’s expectations for the Ceph cluster in production.

The customer uses Commvault as a data backup solution for their entire customer environments.

Wherever the data resides, Commvault provides the backup of the data through a single interface. The customer thus avoids costly data loss scenarios, disconnected data silos, lack of recovery SLAs and inefficient scaling.

For all these scenarios, the customer relies on Ceph as a powerful and cost-effective storage backend for Commvault.

With over 2 PB of backup data and more than 1 TByte per hour of backup throughput, Ceph can easily handle and is ready to grow even further with the requirements in the future.

In conclusion, we were able to clearly exceed the customer’s expectations of the Ceph Cluster already in the test phase.

Working with our partners, we have drawn on our many years of experience with Ceph to create a compact, variable, and highly available hardware/software platform.

This storage solution is targeted at enterprises looking for a reliable and future-proof solution to store enterprise data.

We placed special focus on the flexibility and extensibility of the platform.

deployment scenarios​

For example, the storage platform is designed for the following usage scenarios:

Scenario PoC – Proof of Concept​

The platform can be equipped as a basis for PoC- Proof of Concept and then expanded for production use later.

Scenario HA – High Availability Storage Solution​

Failures of single components, like hard disks, controllers up to whole servers, can be compensated without downtime and with maximum data security.

Scenario GEO HA – Georedundant Storage Solution​

Full data center failures can be safely compensated for during operation.

Scenario replacement of existing systems

Ceph as Software Defined Storage offers several ways to import existing data.

e.g. via NFS, ISCSI, Object Storage (S3, Swift)

Tethering options​

Ceph – “The Future of Storage” offers connectivity on one platform as:

• Distributed File System
• Block Storage
• Object Storage (S3 and Swift)
• ISCSI
• RADOS

Consultation and operation​

Clyso GmbH, as members of the Ceph Founation, with its proven expertise in Ceph, will be happy to take over the tasks from planning to operation of the platform.

Today, Kubernetes is the first choice for running microservices in the public or private cloud. More and more developers and enterprises are building their applications on the modern microservice architecture.

Many of them are using Kubernetes for automated deployment of their workloads and want to benefit from the new flexibility and robustness. We are working on a solution for our customers to simplify and unify Day One and Day Two operations in their operations. With the increasing number of clusters, the management, updating and monitoring should be able to deal with it efficiently.

Since 2018, we have been accompanying Rook.io in its development and had direct exchanges with various members of the project at Cephalocon in Beijing 2018 and Barcelona 2019.

In 2019, we began serving customers in production who use Rook.io to manage Ceph.

Storage Operators for Kubernetes

Rook transforms distributed storage systems into self-managing, self-scaling and self-healing storage services. It automates the tasks of a storage administrator: provisioning, bootstrapping, configuring, deploying, scaling, upgrading, migrating, disaster recovery, monitoring, and resource management. Rook leverages the power of the Kubernetes platform to deliver its services to any storage provider through a Kubernetes operator.

[https://rook.io/](https://rook.io/)

Since 2020, we are now working on improving the automated operation of Ceph with Rook.io. Furthermore, it is planned to have the platform fully audited by various certification bodies.