I've started buying "nicer" things purely for fit reasons, like I've realized that I'm a lean 6' guy with a long torso, so for shirts and especially sweaters a medium is too short and a large is too baggy— the correct size for me in off-the-rack items is a tall medium, and that's definitely not available everywhere.

So I'm extremely happy with a Land's End quarter zip that I picked up recently, and I hope that's a well-made piece that will last a while, but overall I completely agree that mass-produced clothing is a market for lemons; no one can tell what the good stuff is, so it's all assumed to be garbage and priced accordingly.