I use this setup as well. I've written an alias for bwrap which only gives the agent access to the current directory and read only access to the docs.

It has read-only access to my binaries, and I worry which programs I have might be a fingerprinting issue. I've thought about mounting a alternate /bin which using a docker image. (docker export)

Have you managed to setup any firewall or protection for the API keys? I know this is out-of-scope for bwrap.

edit: Maybe https://github.com/rootless-containers/slirp4netns could work