logoalt Hacker News

wolvoleo • today at 8:30 PM • 0 replies • view on HN

Yes especially now that local context isn't so costly anymore. With llama3 it would kill my vram just to have a few thousand tokens. With recent models it's hardly an impact anymore. Not sure how they manage that but it's great. It still takes more processing but I also run an older card that is not very good at prefill.