There are only certain kinds of computing tasks which are amenable to an exponential speedup from quantum computing. For many classical algorithms the best you get from a quantum computer is an improvement by a factor of sqrt(N) by using Grover's algorithm.

The other tradeoff is that quantum computers are much noisier than classical computers. The error rate of classical computers is exceedingly low, to the extent that most programmers can go their entire career without even considering it as a possibility. But you can see from the figures in this post that even in a state of the art chip, the error rates are of order ~0.03--0.3%. Hopefully this will go down over time, but it's going to be a non-negligible aspect of quantum computing for the foreseeable future.