The same agents that could potentially leak your secrets? I would rather not give a hacker a cached session that unlocks the keys to the kingdom.
I'll take security by inconvenience over building what becomes the primary reason for a security incident.
If I already approved it once I already have to assume it could have been leaked. The cache doesn't change much about it if the agent / tenant is asking for something it already has.