It is important to remember that Microsoft created some of this chaos to begin with. Other aspects can be attributed to "the industry didn't understand the value of $x or the right way to do $y at the time". And some of this is "nonsense you deal with when the internet and automatic updates is not yet a thing".

Why did programs overwrite system components? Because Microsoft regularly pushed updates with VC++ or Visual Studio and if you built your program with Microsoft's tools you often had to distribute the updated components for your program to work - especially the Visual C runtime and the Common Controls. This even started in the Win3.11 days when you had to update common controls to get the fancy new "3d" look. And sometimes a newer update broke older programs so installers would try to force the "correct" version to be installed... but there's no better option here. Don't do that and the program the user just installed is busted. Do it and you break something else. There was no auto-update or internet access so you had to make a guess at what the best option was and hope. Mix in general lack of knowledge, no forums or Stack Overflow to ask for help, and general incompetence and you end up with a lot of badly made installers doing absolute nonsense.

Why force everyone to share everything? Early on primarily for disk space and memory reasons. Early PCs could barely run a GUI so few hundred kilobytes to let programs have their own copy of common controls was a non-starter. There was no such thing as "just wait for everyone to upgrade" or "wait for WindowsUpdate to roll this feature out to everyone". By the early 2000s the biggest reason was because we hadn't realized that sharing is great in theory but often terrible in practice and a system to manage who gets what version of each library is critical. And we also later had the disk space and RAM to allow it.

But the biggest issue was probably Microsoft's refusal to provide a system installer. Later I assume antitrust concerns prevented them from doing more in this area. Installers did whatever because there were a bunch of little companies making installers and every developer just picked one and built all their packages with it. Often not updating their installer for years (possibly because it cost a lot of money).

Note: When I say "we" here that's doing a lot of heavy lifting. I think the Unix world understood the need for package managers and control of library versions earlier but even then the list of problems and the solutions to them in these areas varied a lot. Dependency management was far from a solved problem.