alt Hacker NewsI don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
Replies
shric • yesterday at 9:40 PM
Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.