The issue is that so much of night time lighting is wasteful and excessive, not that lighting doesn't enhance the world for humans.
I know for a fact that just putting old school light shutters on streetlights so they don't leak into, say, bedrooms and stay focused on pavement and sidewalks, and dimming them off hours, would be a huge change that doesn't diminish utility at all.
I once saw motion activated streetlights that went from dim to bright as cars approached. It seemed very futuristic, and I wish a lot more places had them.
> a huge change that doesn't diminish utility at all.
It does though, by redirecting funds & human effort that could be used for more lighting.
Here's a counter-anecdote though, to give you hope.
In Bondi in Sydney there's a lighting upgrade underway on the beach promenade; the plan was circulated so I got to take a look. As you hoped they indeed did deeply consider throw angles and what's lit & unlit, and ran a competitive tender to choose among many vendors, settling on one they lauded for their flexibility in offering products that can throw at specific angles and leave specific areas unlit. In the plan published for public comments they specify the exact angles needed with diagrams and everything. So it's possible! It requires public servants to go above and beyond, which does happen, just not all the time and not in all the places. But it does happen!