What I've seen lead to success:
* Arrogance
* Overconfidence
* Schmoozing with the right people
* Doing flashy work, whatever that means in a given situation
What I have seen lead to failure or, at best, being undervalued and ignored:
* Caring about teammates and your future self
* Caring about the end user and the business itself, when it conflicts with something sales, marketing, or a PM want
* Creating resilient, well-engineered systems
It's the same problem as anywhere else. Well-crafted systems are invisible and taken for granted. Saving the day by putting out a fire is applauded, even when you're the one who laid out the kindling and matches. Managers at all levels care about their own ego more than the company, product, or team.
Maybe I just spent too much time with ex-Microsoft hacks.
Overconfidence is a slow and insidious killer
This can applied to a lot of sectors, look at the arts and culture for example
No, you are right
No, the first one thrives because they know how to play politics, the second one fails because they don't know how to play politics.
You described word for word the archetypical engineer, competent technically, incompetent politically. A liability to his team and superiors in a cut-throat corporate environment. That's why they fail, they can't be trusted to not screw their team over to do the right thing.
Early on in my career I couldn't understand why it was always the worst and most incompetent people who got promoted.
Then I realized that it's not their incompetence that gets them promoted per se, it's that if they're employed while being utterly useless and incompetent they have SOMETHING else going on that keeps them employed.
And it's that something else (whether that is politics, brown nosing, nepotism, bullying) that also gets them promoted.