Agreed it isn't necessarily a bad reason. In some cases it's a good reason for failure (like the one you describe).
In other cases it's a bad reason for failure: it's also incredibly expensive to prove your drug works even if it does work for a lot of people.
That's bad! It'd be better if it were cheaper.
Actually counterintuitively, due to a weird drug approval and payor reimbursement policy arbitrage, pharma companies are highly incentivized to produce drugs for tiny populations.
One of my hobby horses is railing against this specific dynamic.