People are not freezing in a cold climate - they have plenty of coats on. In hot climates you run out of clothes to take off - even nudists.
Well I feel cold in winter sometimes even with a coat on. It hurts when I go outside, so I stay inside more, but if I stay inside too much, it hurts.
The point about hot environments is true, but people are not anxious and your body rarely hurts. They are lazy and their minds blank out. It is often too hot to do anything except try to scam anxious northerners and move away from mosquitos.
I grew up in northern Sweden. You're definitely miserable even when dressed perfectly in -15°C!
You're right that once it gets over +30°C or so, you'll be miserable whatever you wear. But there is a large temperature range below that that is wonderful. The Bay Area is almost always in that zone.