If the LLM is running under a role, which it should be, then RBAC can help.