Without disputing the added latency of CGNAT, the v6-only peering fights (not just the infamous Cogent-HE dispute but smaller regional ISPs peering only on v4) means that there are indeed cases where v6 is worse than v4 in practice. Again, nothing inherently wrong with v6 itself, but peering disputes means bad latency on v6, which means that protocols relying on TCP (like plain FTP, SFTP and rsync) really takes a hit in transfer speeds.

Also there are cases where the ISP didn't bother even optimizing their routing in v6. I understand that some ISPs in Asia (and especially in Japan, where it shows up on ordinary customers in terms like MAP-E and VNEs) have separate backplanes for v4 and v6 (some are legacy reasons, some are business reasons). Guess which one is being devoted more in practice (hint: not the one being devoted more by IETF).

Edit: I thought this was just in Asia, but apparently this is also the case in an ISP in UK (https://news.ycombinator.com/item?id=48618403)