At least for transformers, it can be kind of fixed with MOE + NVFP4 for small working set despite large resident size.