This feels like an exceptionally bad tradeoff. HBM is expensive due to the packaging cost and yield problems. Now you're trading the relatively inexpensive but high performance DRAM for Flash while splitting the addressable market into training and inference only hardware.

The low capacity of HBM isn't really a mistake. It's a design decision to keep the bandwidth to capacity ratio high. HBM systems with 96GB of memory tend to have around 3.5 TB/s which is a ratio of 35:1, meaning your theoretical maximum is 35 tokens of inference per second assuming you use the full storage just for parameters.

If you massively increase the capacity but keep the bandwidth the same, you just end up lowering this ratio. Your system is overall smaller, but it also has less performance.

This makes High-Bandwidth Flash an extremely niche product or the equivalent of industrially processing lampante olive oil and mixing into high quality olive oil. E.g you're spending an extreme amount of effort on making a worse product that is only marginally cheaper in absolute terms, but more expensive in terms of price to performance ratio.