Yeah but I don't actually care if some orphaned cache or config file gets left behind if it doesn't take up GBs of space.

Clearing the package receipt database of stuff you want to uninstall is fucking neurotic, I'm sorry, but it just is.