FOSDEM 2024, 3 & 4th of February
We will be at FOSDEM again this year. this year we will also be giving a presentation on one of our own open source projects CHORUS. The topic is the management and distribution of data and the life cycle across multiple object stores. From real life, explain how we used CHORUS to move a productive object storage without interrupting operations.
presentation details
Chorus - Effortless Ceph S3 Petabyte Migration
room: K.3.201 date: Saturday 3th February, 15:30–16:00 (Europe/Brussels) Video conference: k3201
Efficiently migrating petabytes of object storage data between two production Ceph clusters posed a significant challenge with live data being written to both clusters, necessitating a seamless process to minimize disruptions. The migration strategy involved extracting user accounts, including access and secret keys, from the old cluster and seamlessly transferring them to the new one. The synchronization of buckets and live data has been improved by extending and enhancing powerful tools such as rclone, executed in parallel. This migration endeavor not only resulted in the successful transfer of vast amounts of data but also paved the way for the creation of a robust tool named Chorus. Chorus, specifically designed for synchronizing S3 data, emerged as a versatile solution capable of harmonizing data seamlessly across multiple cloud storage backends. This innovative tool is helpful in effective bridging of data between Ceph clusters, demonstrating the adaptability and scalability required for modern data management challenges. Key highlights of Chorus include persistence of migration, execution of migration on multiple machines, rate limiting RAM/network usage during migration.
FOSDEM 2024 - Chorus - Effortless Ceph S3 Petabyte Migration