alt Hacker NewsA lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.