logoalt Hacker News

PrairieFireyesterday at 8:07 PM1 replyview on HN

A future where we carry and manage just one device could be incredible. That said, today, even if iOS weren’t so locked down and more capable of that, I think I’d find myself frustrated. I run on device local llm’s on my iPhone and a heavily quantized 3b parameter model starts to cause the iPhones thermal management to heavily throttle after just a few prompts with light tokens, to the point it’s slower than 1 token per second for inference or response, and the phone gets hot to the touch. Maybe the rumored half iPhone half iPad device could be the eventual platform from which something like this emerges.


Replies

WorldPeastoday at 5:20 AM

perhaps that's what they're developing all these "private compute" servers for. Though I would be less than happy if Apple, the last (relatively) untaken hill of the SaaS enshittification wars were to go down that road. In the meantime I will continue to use my hilariously overpowered laptop as a SSH terminal to the machine I actually work on