1- Costs of operating an larger house are higher than a smaller house 2- Yearly maintenance of a new house is cheaper than an older house (older the house, the greater this gets) 3- Easier to keep a smaller house tidy 4- Mobility/lifestyle issues. 5- Indifference towards resale value
Not everyone wants a giant house, and plenty of people like a small one. It's not the size that matters, it's how you use it.
I don't think anyone is talking about giant houses. Last time I priced a new 1,500 sq ft home they wanted over $800,000. That was an instant nope from me. Another commenter pointed out that their 650 sq ft home would be $400,000 to build new. These are already beyond most people's affordability even before considering the cost of the land.
So when we talk about small and affordable we're talking more like 100 sq ft. Large compared to that is what most would call a modest home.