Outside of the PQ algorithms not being as thoroughly vetted as others, is there any negatives to shifting algorithms? Like even if someone were to prove that quantum computing is a dud, is there any reason why we shouldn't be using this stuff anyway?
they are much more thoroughly vetted than other schemes. They're more thoroughly vetted than elliptic curves were before we deployed them. Much more vetted than RSA was ever.
Practically though, there are some downsides. Elliptic curves tend to have smaller ciphertexts/keys/signatures/so are better on bandwidth. If you do everything right with elliptic curves, we're also more confident in the hardness of the underlying problems (cf "generic group lower bounds", and other extensions of this model).
The new algorithms tend to be easier to implement (important, as a big source of practical insecurity is implementation issues. historically much more than the underlying assumption breaking). This isn't uniformly, e.g. I still think that the FN-DSA algorithm will have issues of this type, but ML-DSA and ML-KEM are fine. They're also easier to "specify", meaning it is much harder to accidentally choose a "weak" instance of them (in several senses. the "weak curve" attacks are not really possible. there isn't really a way to hide a NOBUS backdoor like there was for DUAL_EC_DRBG). They also tend to be faster.
Post-quantum algorithms tend to be slower than existing elliptic curve algorithms and require more data to be exchanged to provide equivalent security against attacks run on non-quantum computers.
AFAIK, PQ certificates are significantly longer than current ones. I don't know exact numbers though.