Because it isn't a free market in the USA. And those that regulate it don't seem to care. Or maybe it is those that have been granted a monopoly do everything they can to retain said monopoly. Things would be different if we actually had a free market
And even with a free market, there's areas where making up the investment would be difficult, because the amount of effort divided by the number of likely subscribers, still wouldn't pencil out for 20+ years. A lot of Americans live in suburbs that are just low density enough that updating the infra to get fiber anywhere near the house is expensive, and then you might have quite a bit of fiber specific to that one subscriber. The difference in how much infrastructure you need vs a city is substantial.
Yes, the duopolies are due to regulatory capture in the US.
A lot of ISPs need permitting that they will never get in order to enter a new market/location.
Free market enthusiasts' reasoning is literary the same as Communists': when their grand theory fails to deliver its grandiose promises, it's nver because their believes where nonsensical, but because “it isn't real Communism/free market”.
Some markets just inherently turn non-free very quickly when left unsupervised. Especially infrastructure markets.