alt Hacker NewsYou can already run it locally its just not the same.
It is still slow, a lot slower than what you are used to with claude and co.
And as soon as you increase context size, your memory requirements jump.
Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
And even a used 3090 is apparently now between 1-2k.
Replies
blahblaher • today at 3:01 PM
yeah, but otoh... f* Anthropic and OpenAI
> It is still slow, a lot slower than what you are used to with claude and co.
That really depends on the model, I run a few models locally. All at speeds comparable to or faster than Opus.
In general we haven't reached the ceiling for what performance we can get out of consumer hardware. As evidence by FreeToken which hasn't even added MTP/speculative drafting support yet, which will add another boost.
> Then when it runs for 30 minutes for something claude needs 5, your device will get hot.
I doubt the timing differential here, but even still I run my 3090 pretty heavily with inference workloads and it stays cooler than when I use it for gaming.
> And even a used 3090 is apparently now between 1-2k.
Yeah I guess the price went up significantly in the last couple months, used to be hovering around 1k. 3090 isn't the only option though.