alt Hacker News12 tok/s and almost instant response on M1 Max Mac Studio (with faster SSD than laptops) are impressive – gives hope that large models may run locally from SSDs instead of memory.
12 tok/s and almost instant response on M1 Max Mac Studio (with faster SSD than laptops) are impressive – gives hope that large models may run locally from SSDs instead of memory.
Thanks for sharing! SSD read speed is the biggest limiting factor here, unfortunately