alt Hacker News> On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds
The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.
Replies
akmarinov • today at 5:25 AM
The author put in the numbers, but maybe you didn’t read them.
45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio.
Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash
➕ show 1 reply
The difference is memory bandwidth. The M5 Ultra that's coming out on 22nd September can do 1,200GB/s. The M5 Max you can buy today only has 614GB/s.