alt Hacker NewsMy initial implementation of MTP I have here in my own (DGX Spark specific) custom runtime brought it up from ~12 tok/sec without MTP to ~16 to ~20 with; depending on workload.
It's not world changing, but at those speeds I'll take anything I can get.
(The ngram embeddings in this case are paged to/from disk which seems to cost ... basically nothing).
It should be faster, Q6 on Ryzen 395 using Vulkan llama.cpp is about 22 tokens/s with no MTP, and I'd expect the Spark to be 20% faster or something in that neighborhood.