Because democracy is "one person one vote", not "one dollar one vote".
Around the same time Citizens United was decided, we also got McCutcheon v. FEC, which invalidated campaign contribution limits basically completely. If we take the logic of Citizens United at its word - that money is speech - then letting someone drop billions of dollars to change an election is like firing a sonic weapon at a bunch of protesters to silence them. So, right off the bat, we have a situation where protecting the "speech" of the rich and powerful directly imperils the speech of everyone else.
But it gets worse. Because we got rid of campaign financing limitations, there has been an arms race with campaign funding that has made all speech completely, 100% pay-to-play. We have libre speech, but not gratis speech.
This isn't even a problem limited to merely political speech. Every large forum by which speech occurs expects you to buy advertising on their own platform now before you are heard. If you, say, sell a book on Amazon or post a video on TikTok, you're expected to buy ads for it on Amazon or TikTok. You are otherwise shut out of the system because discovery algorithms want you keep you in your own bubble and you're competing with lots and lots of spam.
But it is still one person one vote. Money doesn't allow you to buy votes, but it does make it easier to persuade them. Freedom of the press has always guaranteed you the right to print or otherwise publish what you want, but it never said everyone will have the same amount of printing presses or the same amount of ink. Freedom of speech does not guarantee you an audience.
You think you are reducing the influence of the rich, but you are actually just raising the price of entry. A millionaire can donate to a PAC and buy TV ads, but a billionaire can buy or start a newspaper, TV station, or social media network. What are you going to do then, tell the newspapers what they are allowed to print?