The math stays difficult after basic discrete concepts and gets more difficult as you go. :)

It's straightforward to get yourself to a place where you can do cryptographic things and feel somewhat comfortable with what's happening. Truly understanding it to the point where you can reason safely about it is deceptively harder.