alt Hacker NewsI've often wondered why there aren't any (that I'm aware of anyway) Linux hierarchical storage managers (HSM). I realize they are typically used in big iron with things like tape as the base level of storage, but especially when it looked like Optane was going to be a thing it seemed like spinning<->SSD<->Optane<->RAM could be a huge performance opportunity. Though, I guess that's a "layering violation"... Honestly, at this point we've gone almost entirely SSD except for our ZFS backup servers, and that layering violation already has an SSD and RAM cache in it...
Replies
Schlaefer • today at 7:02 PM
RAM cache is already layered in front of persistent storage. When it comes to easy hybrid SSD/HDD setups bcachefs was close to deliver, but then things jumped the shark.
When you're running ZFS (or probably some other clever layering violation like bcachefs) on heterogeneous workloads, you will quickly run into the limits of built-in assumptions. At least that's my experience:
- ARC may require manual tuning for size and metadata size to accommodate page cache, in-app caches, file count etc
- L2ARC and SLOGs are very obviously useful for some clear workloads, but have some cliffs you don't expect, such as ram required for L2ARC mapping, unexpected cold/hot/boot behavior
- Reasoning about reliability becomes harder the more tiers you have
Just imagine, to pick a simple example, you're running postgres, thousands of sqlite3 dbs and one huge duckdb (larger-than-ram) on a single such system - it would perhaps manage to adapt to one such workload at a time, but would it balance all three at the same time?
So much as I'd love to see this direction, I can also imagine the pains coming with it.