In European cities this is mitigated by having low-floor buses and stops with level boarding to support mobility scooters and wheelchairs. There are also dedicated taxis available for people with disabilities (possibly subsidised). Over a long term this is also a self-regulating problem. Elderly people and services/businesses for them take into account availability of public transit when choosing properties.
Buses are mass transit. The real goal isn't serving poor people, but moving people with higher throughput than it's possible by cars individually (a single bus fits ~50 people). If you make bus lines slow and fail to attract significant numbers of passengers by forcing buses to serve every whatabout case, you're making them fail at their primary goal.
You can't make half-pregnant public transit. If you have a congested city, and just add nearly empty buses sitting in traffic and blocking lanes at every intersection, it will be strictly worse for everyone. OTOH if you can make buses an attractive option, then each bus can take 30+ cars off the road, leaving room for dedicated bus lanes, more buses, resulting in faster and more regular service.
"If you have a congested city"
I would agree with and extend your remarks that we also have problems where traffic patterns and geography don't match political boundaries and transit is traditionally locally run and locally budgeted.
So in the USA you end in scenarios where it takes 20 minutes to drive 20 miles but a bus would take four legs with three transfers across three separate city bus companies, figure at least three hours each way. And again, as per your "mass transit" you can't expect taxpayers in my city to provide a special bus run into my neighboring adjacent city much less the city next to that one.
This results in people being very happy indeed to pay the financial and environmental costs of car ownership to avoid sitting in a bus for six hours of daily commute.
There are also interesting social issues; if you're late its a personal failing, even if you take mass transit. I recall a friend at work getting fired because the bus was late too many times. Oh well, should have bought a car. The feeling of not being in control is further worse due to crime rates. No one will sneak up on my wife and stab her in the neck in her car, but it certainly happens on buses and no one cares if it happens depending on local race relations. None of the other passengers on the bus even cared, for racial reasons. Its pretty messed up here.
Its easy for the public in general to advise others to do inconvenient or career ending or life threatening activities, to "save the planet" or whatever, but I wouldn't do it, and I'd certainly never let my wife or kids do it, so we own cars and avoid public transit at all costs. Not taking that advice as been pretty nice so far.