You need to secure the account an LLM-based app runs under, just like you would any user, AI or not. When you hire real people, do you grant them full privileges on all systems and just ask them not to touch things they shouldn't? No, you secure their accounts to the specific privileges they need, and no more. Do the same with AI.
You'd be surprised. I've worked at multiple startups where employees were given prod access with zero oversight on day one: AWS, sudo access, database passwords, everything. The one startup that didn't do that never launched. Occasionally there were accidents: wrong branch deployed, bulk updates to DNS taking down most of the site, etc.