I use podman for things like this, works perfect until you want desktop applications but you can hack it about a bit to work fine with pipewire and Xephyr and you have. I feel like a lot of these desktop container systems are horrible and are quite hostile to configuring in the way you want around permissions and such and podman or docker does a better job.