All modern cryptography is based on problems with some mathematical structure. With enough structure it's true, it does weaken security, and many cryptosystems are broken by exploiting the underlying structure. So I suppose the only solid answer here is that attacks haven't been discovered yet despite many smart people trying.
Heh, I didn't realize that you were the Jeremy Kun that wrote the piece that gave me a bit of enlightenment (that I mentioned in the edit to my comment above). Thank you for writing that. It was helpful.