People assume that accessibility is all about some small minority of less abled people who can't "read good", but it's a broad category that affects all users. If you build following the guidelines then you end up with a quality product that can be used by people who stumbled upon it while doom-scrolling instead of enjoying their beach vacation. The best analogy I heard was about drop-kerbs/curb-cuts... people wonder why we're catering for a small minority of wheelchair users everywhere and then they have a kid (or wheel luggage from the airport) and realize how great they are.
Yup, accessibility is literally about broadening the population of people who can use your software. It's often associated with affordances for the less-abled, but that's just a subset of accessibility. I don't get the hostility! Just a guess but maybe 1 in 50 or so developers I've worked with in the past didn't just "not care" about accessibility but were outright hostile to it, as in affirmatively "We should not spend time working on this!" Bizarre.