It's indeed thought to be really hard to factor multiples of two massive primes, but we don't know that for certain. See https://en.wikipedia.org/wiki/Integer_factorization -- "Unsolved problem in computer science: Can integer factorization be solved in polynomial time on a classical computer?". In fact, we don't even have a proof that this is more complex than multiplying the component primes.

And of course, many cryptosystems are reliant on primitives with much less algebraic structure.

But to be clear, LLMs would presumably break these cryptosystems by building new algorithms and writing code to break them, not by "intuiting" their way to a specific private key.

an LLM would likely just converge on something like a shared prime GCD attack; basically finding private keys somewhere in their training set and then hoping that whatever keygen algo was setup incorrectly and used a shared seed.