I switched everything over to rootless podman a year or two back. Some containers ended up with permissions issues when trying to read their old data - caused by being run with a different UID. This was really the only problem I ran into, but I would have had the same issue switching from rootful docker to rootless docker.

Absolutely zero regrets, would never go back.