You have to break both algorithms. One of them is quantum-safe if it's secure, but it could also be completely insecure like supersingular isogeny was.