alt Hacker NewsProbably long term each dev gets their own GPU and runs a model locally I expect. Seems like a more sustainable approach, even if a local model is not absolute SOTA.
Probably long term each dev gets their own GPU and runs a model locally I expect. Seems like a more sustainable approach, even if a local model is not absolute SOTA.
GPUs are much more efficient at parallelizing requests for LLMs so it's going to much more efficient to centrally host. Maybe big companies it would make sense to get their own though.