I'd like to see this problem solved in such a way that your group memberships are transitively applied to your access to agents. In other words: you shouldn’t be allowed to talk to an agent at all if, at a given point in time, that agent is or was in a group that you are not or were not in at that time. Does that make sense? I don’t know how practical this restriction is, but letting the AI manage such permissions on its own seems reckless to me.