The trick is to reboot into recovery partition, disable SIP, then run OmniDiskSweeper as root (as in `sudo /Applications/OmniDiskSweeper.app/Contents/MacOS/OmniDiskSweeper`). Then you can find all kinds of caches that are otherwise hidden by SIP.
It shouldn't be this hard to clear unwanted data from my own computer
Hmm, Full Disk Access perm is not enough?