All these very large models are sparse mixtures of experts (SMoE), where typically only 2-3% of weights are active for a given forward pass.

Usually there are some static experts in each attention layer, plus a whole bunch of dynamically-routed experts. Typically some form of LRU cache is used for the dynamic experts.

So you can keep only the static expert(s) and routing layers in memory and then stream the dynamic experts from SSD, at a modest inference speed cost driven by the latency required to load layers from SSD when they aren’t already in the memory cache.

The LRU cache works reasonably well because a given task will often hit the same experts on each iteration of the forward pass.

For example, on a Mac Studio, Qwen3.8-flash-next (180b total parameters) with 50% of experts in memory and n-gram offloading has about 1/4 to 1/3 the inference speed of everything in memory. That requires 32GB of vRAM for the weights (plus a little extra for context, obviously)