alt Hacker NewsThis is the real problem with LLMs. There is no way to separate code from data. At best, models could be trained on tokens that indicate untrusted data coming in. But then the untrusted tokens could also be messed with.
I've wondered if it would be possible for there to be two input streams: 1, for prompt, 2 for untrusted data. But I suspect that transformers would still only optionally decide what each one was for. So it would still be a prompt level suggestion, rather than a hard and fast rule.
LLMs should never be trained on restricted data of any kind, as we have seen that they are able to reconstruct their training data. The idea that they could be trained on private/restricted/copyrighted data and that was ok because there wouldn't be redistributing that data should have been killed 3 years ago.
Embedding vector indexes are how we separate code from data. Anything that is not for 100% unadulterated public access should be behind a traditional access control system. RAG search is not magic, it's just a SQL query of a manually created index. It absolutely could have access control built in. It's been out of laziness that it has not.