I think it's because the people who want the smaller phone don't just want the smaller phone to the exclusion of everything else.
Some want a flagship model, some want a mid range, some entry level... all in small form factors and they won't compromise in order to settle for the one small model that's available.
You are describing me as well, but our market size is negligible compared to to 50% of the population that throws this thing in a purse.