I don't know, does new housing or municipal services get built in anyone's literal backyard? So it's not Your or My Backyard, really.
NIMBYism has always been about nosy people obstructing progress.
Casting shadow on their backyard. Bringing noise to their street. Ultimately, lowering the value of their property.
The key problem of US housing is that a house is seen as an investment vehicle, which should appreciate, or at least appreciate no slower than inflation. Keeping prices high and rising can't but go hand in hand with keeping supply scarce.
It should really be called NIYBY-ism.
Literal NIMBY-ism, where the backyard is one's own property, is just straightforward property rights. They want to control other people's property and tell them what they can and can't do with it. That's basically communism.
It's actually about people not wanting the largest investment of their life to change in ways they don't like.
"in my/your backyard" is a very old and pretty common idiomatic phrase that refers to the general area you live in (neighborhood, town, city, etc).