QLC - Guidance on QLC drives
Problem
Guidance is needed regarding whether it is a good idea to use QLC SSDs.
- Does a QLC-based OSD work well without separate DB?
- In general, do you think that it makes sense to have a closer look at QLC?
- Is it different when price is the main priority?
Some background information:
- Assume an average object size of 1 MiB.
- Assume the use of EC4+3.
- Assume low DWPD (drive writes per day).
Specifically, provide guidance about the Intel D5-P5316.
Solution
The D5-P5316 is an interesting drive as it has a very large 64K IU (Indirection Unit) size versus the standard 4K IU size in most TLC (triple-level cell) drives. Ceph can be configured at the BlueStore level to have a similar minimum allocation unit size to match this, but there are some specific things you should be aware of when using a drive like this. When you perform a write that is smaller than the IU size, the drive will suffer significant write amplification and potentially space amplification. All writes smaller than 64K will behave like a 64K write and even writes larger than 64K will be rounded internally up to the next IU size multiple. This can have a pretty dramatic performance and endurance impact when doing small metadata appends to the WAL (write-ahead log), but also can be really important if you are doing erasure coding.
Modern, post-D5-P5316 drives use an 8KB or 16 KB IU (indirection unit) size, which means that small writes don't have the same impact. Modern QLC drives are also faster overall.
Does a QLC-based OSD work well without separate DB?
Database writes tend to be pretty large and potentially can work well with QLC drives, but the WAL is a different story. The WAL will have a large number of small metadata appends performed for every write IO. QLC drives tend not to be great at this. When Intel introduced the D5-P5316, the idea was to pair it with optane for metadata workloads. The idea in Ceph was that at least the WAL, and maybe the DB would be placed on optane while data would be placed on their QLC drives. Now that optane is no longer being made, the industry is moving toward using high speed SLC flash drives to fill that niche. Having said that, for very read-focused clusters, it may be possible to get away with only using QLC drives, though write small write performance is likely to suffer. It's still likely to be faster than HDDs for small writes, but not nearly as fast as TLC based systems
In general, do you think that it makes sense to have a closer look at QLC?
Yes. Even with QLC's trade-offs vs TLC flash, there are many scenarios where it will remain significantly faster than HDDs. QLC drives are excellent for read-heavy workloads and can still be significantly faster than HDDs for write-heavy workloads that have large IOs. The area where they really suffer is for small writes. Depending on the price, they definitely may be worth considering vs HDDs.
Is it different when price is the main priority?
Generally speaking, it's difficult to beat HDDs purely on price. Without knowing the specific vendor agreements that customers have in place, we expect even QLC flash to have a significantly higher upfront cost versus HDDs at this time (Q2 2024). Having said that, the long term benefits of better performance, faster recovery times, and easier data migration are hard to quantify. We've serviced HDD-backed clusters that would have required significantly less emergency support time (for a customer that was paying hourly) had they been flash-based.
Regarding the information you provided here are some other thoughts:
Veeam is in use. Assume an average object size of 1 MiB. Assume EC4+3. Assume Low DWPD.
One thing we know about Veeam is that it tends to break data up into smaller than expected chunks. We've heard from other customers that in some cases they see average object size as small as 300KB despite storing significantly larger files. It may be worth double checking the average object size just to make sure it's really 1MiB.
Assuming the average object size is 1 MiB, using EC4+3 means you would have
256*(4+3) = 1792MiB of raw data spread over 7 256KiB chunks. These are large
enough that you shouldn't typically have chunks smaller than the IU size, but
you may see some write amplification if you end up just over a boundary. For
example, a 1.01 MiB object would take up: 320*(4+3) = 2240MiB of space which is
a 25% jump vs a 1.00 MiB object. This effect gets worse the higher the k+m
erasure coding value you use.
If low DWPD is acceptable, that generally implies a read-oriented workload. QLC drives typically excel at this, and the write-amplification effect and slow performance of WAL writes to the QLC drive may be more acceptable. A good friend of ours in the Ceph community runs this way and has been quite happy using QLC drives.
One final comment. Intel sold their NVMe business to Soldigm who has continued to release QLC products. They now have the D5-P5336 out with a smaller 16K IU size. While we have not yet evaluated this drive with Ceph (We are in discussions with Solidigm to work together to test these drives), the smaller IU size may result in better small write performance.
Some product links: https://www.solidigm.com/products/data-center/d5/p5316.html#form=U.2%2015mm&cap=15.36%20TB https://www.solidigm.com/products/data-center/d5/p5336.html#form=U.2%2015mm&cap=15.36TB