Seems to me you are focused on money rather than time here. However, people using these road are not in alignment with that metric. What’s important is not the cost, but the value. Why choose a toll road over a public path from A to B? It’s all about saving time—that’s the value the transaction provides those users.
If a toll road becomes public, its value goes away because traffic increases on it, eliminating any benefit of traffic reduction that provides time savings that the gate keeping of the charging a toll provides. Also this notion of “everyone now bears the cost of the road” creates damage. That cost now hits the folks who don’t want to use it currently because they do not see its value. All you have done is hurt both the drivers of the road and the drivers who do not use the road by “sharing the burden” and making it public.
I'm not sure money and time are cleanly separable metrics for this in that changing one changes the other, which is why it's not as simple as looking at the toll roads' dynamics in isolation. If a region spends more on transport infrastructure then the average transit time is going to be less (and conversely the opposite) for pretty much any method of buildout except fraud. The notion of "everyone now bears the cost of the road" causes the roads to be optimized towards the average public good. This is not to say it's without tradeoff to anybody at all - just that it's geared towards the best tradeoff for everyone as a whole. The notion of "those able to bear the cost and extra overhead of the toll road bear its cost" certainly still causes a reduction in time for those willing and able to pay, but only for the able who now have no interest in their public infrastructure funding duplicating a path they already have for the common good.
That's where the shift in burden to the poor so the rest can have shorter commute comes from. If everyone had the same opportunity cost to use the toll road then it wouldn't have the shift in burden as much as a pure shift in utility. Of course it doesn't have the same opportunity cost, so who benefits from the toll road is more slanted than who benefits from the public road. Whether or not the shift of burden is acceptable/ideal is a matter of opinion on public policy, but it's there.