logoalt Hacker News

nsagent • today at 6:12 AM • 0 replies • view on HN

That's a truly ingenious and scary investigation.

  Zero Data Retention is supposed to mean something. In practice, these prompts were retained, shared with other companies, and read by people.
The first bit of advice is on point:

  Host it yourself, if you can afford it. This is the only option where nobody else gets to read your prompts, but the hardware isn’t cheap. A Mac Studio with 512 GB of memory starts at $22,299, NVIDIA’s DGX Spark now costs $6,950, a DGX Station starts at about $105,000, and an eight-GPU B300 server lists for $570,000 or more. For most teams, this only makes sense for the most sensitive work.
Despite being an NLP researcher, I never used LLMs for coding until this summer. Glad I exclusively used locally hosted models. My M5 Ultra 512GB just arrived and I'm hoping it turns out to be a wise decision for my startup.