I have this working on a branch of my https://github.com/rdaum/eider (for DGX Spark)

nVME paging the n-gram table (in BF16 for now).

Still working at it. Prefill sucks still but decode is about 12 tok/sec and the model weights fit nicely in the 128GB Spark memory in nvfp4 quant while paging the ngram stuff from disk.

(EDIT: merged to main. 80tok/sec prefill, 12 tok/sec decode, ~80GiB resident, the rest paged)