"Nothing" prevents it other than the fact that an agent doesn't really have memory and has a pretty limited context, hallucinates information, mistakes metadata with data, and so on.

The path forward is always one that starts from the assumption that it will go wrong in all those different ways, and then builds from there