Yea I ran into the exact same issue. My workaround was buying a silicone band that wrapped around the top of the set to help as a sort of "2nd layer."
It isn't perfect, but it makes them wearable.
Pretty incredible oversight by a company that focuses so much on "design."
The bands sell pretty well on Amazon from what I can see so this isn't an isolated issue.
Interesting. Echoes the failure of the original Vision Pro knit headband, Version 2 of which is much better — but it took 2 years to appear!