It's a free country and you can dress as you please.
But people will judge you by how you dress, and you will miss opportunities as a result, and you'll never know that this is happening.
As I mentioned earlier, people do react to me differently depending on how I dress. And I've known many people who align with your views on this, and they've all wondered why opportunity passed them by (or they realized they needed to change).
Can I ask: suppose you were charged with a crime. Your lawyer showed up in track shorts. Would you get another lawyer? I sure would.