It is like it is because:

* It was designed by people who didn't have the full picture and were missing representatives from hardware vendors, small businesses, home network admins and a bunch of other people that will be affected by design.

* It was designed by people who didn't consider the cost of migration and the amount of work that would require (see previous point).

* It was designed by people who lived in an ivory tower of "noone will run dual stack for a long time", "everyone will love to run two completely separate network designs".

* It was designed on a premise that end-to-end, fully accessbile devices are something we actually want and won't cause privacy issues.

I think it should be a study material on how standards and designs by commitee can go wrong if they're not headed by people with extensive experience across the industry with enough authority to push for good solutions.

IPv6 tried to do too much (just like many software "let's refactor this legact code") and was done by people who didn't consider all perspectives and costs (again, like many less experienced architects trying to rewrite legacy software).