Why senior engineers let bad projects fail

Reminds me of one of my managers who said, “Sometimes, you have to let people fail.” It does take a lot of energy to keep some people afloat. My hope has always been they learn to swim as it were, but sometimes it’s just effort better spent elsewhere.

I know one project did not have my involvement and couldn’t have succeeded without my knowledge. They were so bad they would work in questions casually to their actual work.

I started avoiding all of them when I found out management had been dumping on my team and praising theirs. It’s just such a slap in the face because they could not have done well and their implementation was horrible.

I think this advice is pretty apt for small to medium sized companies. We're all invested in the company succeeding, but you don't want to become known as the person that always says "no".

At large companies, I've rarely found a reason to speak out on a project. Unless it has a considerable effect on my team/work (read: peace of mind), it just doesn't make sense to be the person casting doubt. There's not much ROI for being "right".

If you manage to kill the project before it starts, no one will ever know how bad of a disaster you prevented. If the project succeeds despite your objections, you look like an idiot. And if it fails - as the author notes, that doesn't get remembered either.

As a senior IC, the only real ROI I've found in these situations is when you can have a solution handy if things fail. People love a fixer. Even if you only manage to pull this off once or twice, your perception in the org/company gets a massive boost. "Wow, so-and-so is always thinking ahead."

A basic example I saw at my last company was automated E2E testing in production. My teammate had suggested this to improve our ability to detect regressions, but it was ultimately shot down as not being worth the investment over other features.

A few months later, we had seen multiple instances of users hitting significant issues before we could catch them. My teammate was able to whip out the test framework they had been building on the side, and was immediately showered with praise/organizational support (and I'm sure a great review as well).

> It’s important to point out that for much of the lifecycle of a project, whether it’s “bad” is highly subjective.

I can’t emphasize this part enough.

I’ve been part of some projects where someone external to the team went on a crusade to shut our work down because they disagreed with it. When we pushed through, shipped it, and it worked well they lost a lot of credibility.

Be careful about what you spend your reputational capital on.

Excellent advice for the 'House of Cards' politics of big tech, but it’s essentially corporate pacifism.

In any other setting you can't afford to watch money and motivation burn just to stay 'politically solvent'.

(Lalit is very good at fitting complex corporate dynamics in a single blog post though.)

I disagree, and I think this advice can be actively harmful. You shouldn’t ignore a problem when you’re in a position to help. At the same time, you also shouldn’t take on the emotional burden of other people’s projects.

If I see something heading toward failure, I let people know they may want to consider a different approach. That’s it. There’s no need to be harsh or belabor the point but it’s better to speak up than to quietly watch a train wreck unfold.

The dynamics are exactly right, but I would say engineers can't "let" "politically bad" projects fail, because fixing that is, in a very real sense, above their paygrade. That responsibility lies with the executives.

The engineers' role should mostly be as technical advisors, i.e. calling out bad projects for technical reasons (UX, architecture, etc.) But even the seniormost engineers do not have the corporate standing, let alone political cachet, to call out or fix political issues (empire building, infighting between orgs, etc.) They can and should point out these conflicts to leadership (very diplomatically, of course) but should bear no responsibility for the outcomes.

However, as an engineer you should ABSOLUTELY be aware of these dynamics because they will impact your career. Like when the project is canceled with no impact delivered.

The example given of the latent turf war between the product and platform teams might have been avoided via a very clear mandate from senior leadership about who owns what exactly. This would probably have involved some horse-trading about what the org giving up its turf gets in return. (BTW if you've ever wondered "Why so many re-orgs" this is why.) That this didn't happen is a failure on the execs' part.

As an aside, I know this happens in every large company, but somehow it appears to be a lot more common at Google? Or at least Googlers are more open about it. E.g. I observed something similar on that recent post about lessions from Google: https://news.ycombinator.com/item?id=46488819

If one person thinks this way, many more do. This is typical in large organizations, especially government institutions, because expense of running entire teams at massive costs for no reason is not born by the team but by someone with a much larger budget that has more money than care or completely wrong incentives (the more people I manage, the more important I am, type of orgs). This is organizational gangrene described from the inside and partly how or why it happens. If you are leading an organization and reading this - figure out how to measure and prevent it.

Great analogy.

I've never worked at a company as large as Google but in my experience things can be a little more optimistic than the post. When earn enough trust with your leadership, such as at the staff/architect level, you'll be able to tell them they are wrong more often and they'll listen. It doesn't have to be a "$50,000 check" every time.

That leads to a very important question - Why doesn't leadership always trust their engineers? And there's a very important answer that isn't mentioned in the blog post - Sometimes the engineers are wrong.

Engineers are extremely good at finding flaws. But not so good at understanding the business perspective. Depending on the greater context there are times where it does make sense to move forward with a flawed idea.

So next time you hear an idea that sounds stupid, take a beat to understand more where the idea is coming from. If you get better at discerning the difference between ideas that are actually fine (despite their flaws), versus ideas that need to die, then you'll earn more trust with your org.

One thing this blog probably misses is what happens when we suggest something to fix a project’s trajectory, they ignore it, but eventually they silently adopt the suggestion and improve the project.

This results in a net loss of ROI: we risk being seen as a negative person, while no one acknowledges our goodwill or instincts, because the project lead presents the improvement as their own win through a strategic change of plan.

More succinctly:

* Know your audience. Saying things they are unable to hear is a waste of energy.

* Choose your battles carefully.

The flip side:

* Trust your gut

* Speak authentically and with an aim to help (not convince)

* Don’t be overly invested or dependent on the actions and reactions of others (can be hard to do if someone has power over you)

Balancing these things is something I’m learning about…

Definitely a big tech thing I don’t miss. At a startup everyone is trying to make the company succeed vs pet projects, so giving advice about architectural decisions or helping fellow engineers with areas you have more expertise in is often welcomed. There are always pros and cons, but that type of culture is so much more fun. Even on hard days I love working with people who want to help each other.

> Your attention is finite, but the capacity for a large company to generate bad ideas is infinite. Speaking from experience, getting too involved in stopping these quickly can make you very cynical about the state of the world. And this is really not a good place to be.

This also applies to the capacity of the industry to generate bad (and evil) ideas.

Now that we're one of the biggest-money fields, there is no end of people thinking/behaving badly.

You'll wear yourself out, calling out all of it.

For example, I fled cryptocurrency entirely when it got overrun with bad faith. But I don't intend to flee AI, and so will have to ration the criticism I have for abuses there.

> The nuclear option is [...]

BTW, be careful in what context you use this idiom. It doesn't always translate well outside the US. (I realized this as soon as the words came out of my mouth, under perhaps the worst possible circumstances.)

The bank account analogy (or "social capital") ignores distinctions between spending and investing, and how these depend on culture.

If you are simply the wrong person in a "toxic" culture, there is no action that can increase your social capital. In a well-functioning culture, constructive criticism would be investment, rather than spending.

This has some crossover with ageism. In the last years in my org all the senior staff has been put to pasture doing either nothing or confined strictly to whatever they were already doing.

Some part of it is that we are perceived as lazy obstructionist naysayer dinosaurs when we point out any flaws in new projects as the article warns. But the rest is that because some of the elders were effectively semi-retired and doing little, anyone over 40 has been uncritically dumped with them.

So we keep the lights on while all the new shiny stuff is given to fresh juniors that don't ever push back and are happy to say yes, but also can't do it alone, and are lost at sea.

So they don't get anything shipped while we keep polishing our legacy turds and wince every time we accidentally get a glimpse of what they are doing.

Nice article. I like the influence bank account analogy. I have worked with a guy who would go into influence debt and I think it hurt his career. He would almost always be right; very sharp, but he would block people and get on their nerves very often, eventually moving to another team.

Are people here reading the article comfortable sharing it, or similar articles, with their teams? I can't do it, and i'm not really sure why.

The problem tends to be timing. By the time you hear about a project, it's often been approved by multiple layers of management, senior engineer signed off on design, etc.

You might be able to get the engineers to tweak the design, but actually getting it canceled can be hopeless. You'll get told the CEO approved it.

At this point I will generally only stick my neck out to criticize a project, decision, or initiative if one of the following is true:

- It will adversely affect me directly (e.g. cause me to get paged a lot)

- It will harm users or other people outside of the org (various kinds of externalities)

Otherwise it's the company's problem. (Of course, I'm generally happy to give advice and critiques if asked.)

> In large companies, speaking up about what you see as a “bad project” is a good thing. But only in moderation. Sometimes the mark of seniority is realizing that arguing with people who won’t listen isn’t worth it; it’s better to save your counsel.

Knowing something is wonky and knowing how to fix something wonky effectively without pissing anyone are completely different level of tasks though.

Knowing things is bad only requires knowledge of the product itself. But fixing it requires understanding of the whole infrastructure and members around the project.

An outsider can't do it. And the insider don't necessarily think the project is bad from his perspective. You would have to argue with him to convince him the project is bad. Which really don't bring any value to the outsider themselves. And it can even be harmful.

When I read the title I wondered “does this really happen?” And then I read a few lines and remembered “yes, I definitely did this all the time”.

The reality is that a lot of people put ideas forward solely for the purpose of getting attention and trying to get promoted. A lot of those ideas are full of hope and enthusiasm and lacking in fundamentals. But to shoot it down makes you come across as a nay-sayer, or a Debbie-downer. Even though you are sure it’s going nowhere, the reality is that it’s a lot easier to let the market prove you right.

Less hurt feelings and interpersonal drama that way. Seems wasteful, but at the end of the day you got data and learnings that you didn’t have otherwise. So hey, silver linings.

I've seen another type of "let projects fail" in my career done by middle managers in a large project. Essentially it takes the form of them saying "the larger project we are working under is probably going to fail. When it does, I want our component to be useful for whatever comes next". And, the surprising thing is that this often worked. The project itself fails, but most of the work done on it still ended up being used.

I have to question the judgment of the manager talking shit about another team and its leader to a junior engineer. Going and looking at the author's LinkedIn history (it's available via his About page) makes it pretty clear that this was happening within Google.

I think it speaks poorly of their manager's professionalism, and what sort of behavior they consider to be acceptable with regard to colleagues.

Its almost never correct to rip on a project from a distance. Only two things can happen, one, you are wrong, and the project succeeds. This is a personal catastrophe for your career at that company. Two, you are correct and the project fails. Its rare this will get you enough credibility to make the risk worth it. There are always others that will show up and dogpile as if they "knew" the entire time themselves. You need to be consistently correct about failure to get truly noticed, but then it asks a lot of questions. Why are you still working there? Why don't you have enough influence to prevent it in the first place? "I told you so" rarely accomplishes anything good.

Learnt from experience that when you can foresee a project failure and you don't have any power, stay quiet and start applying for new role before the blame game, retros and political chaos. It's not worth it, you're just an employee and it's not your company.

Companies need ways for individuals to bet against projects and people that are likely to fail. So much of the overhead in a large organization is from bad decisions, or people who usually make bad decisions remaining in positions of power.

Imagine if instead of having to speak up, and risk political capital, you could simply place a bet, and carry on with your work. Leadership can see that people are betting against a project, and make updates in real time. Good decision makers could earn significant bonuses, even if they don't have the title/role to make the decisions. If someone makes more by betting than their manager takes home in salary, maybe it's time for an adjustment.

Such a system is clearly aligned with the interests of the shareholders, and the rank-and-file. But the stranglehold that bureaucrats have over most companies would prevent it from being put in place.

In short, you have to choose the hill you want to die on.

reminds me of a friend who worked in japan (an american). something was wrong in a project and nobody would say anything. Turns out he was the one to say something. Japanese culture won't speak up, but a foreigner can make "mistakes" like speaking uncomfortable truths, which breaks the logjam.

This article is very wise and applies equally to marketing and other endeavors within the corporation.

> Manage influence like a bank account

I often use the term "social capital." You have to be careful with how you spend it.

Ive done exactly this.

Upper management agreed to geoIP blocking of the app, without consulting engineering. Why this matters is that GeoIP blocking is at best a whack-a-mole with constantly updating lists and probabilistic blocklists. And is easy to route around with VPNs.

The verbiage they approved was "geoblocking", not "best effort of geoblocking". Clients expected 100% success rate.

When that didn't work, management had to walk that back. We showed proof of what we did was reasonably doable. That finally taught upper management to at least consult before making grandiouse plans.

Good career choice: Don't join teams working on bad projects.

There's a good chance that when inevitably fail, the team will be laid off.

So beware of shiny new projects until they've proven themselves.

Very wise article

As technical people we tend to have a technical outlook to this. However, after a certain threshold - say $1M - these projects become political things rather than a simple technical issue.

From a creator's standpoint, a software project exists to solve a problem - or at least make the lives of the software users easier. But the moment a company bigwig clique decides to make money out of company, "bad" projects pop up.

To my chance, I experienced this for three times. The signs are nearly the same. The company has a lot of workflows - usually handled by excel and/or internally developed apps that actually reflect those workflows. Then comes the buzzword team proclaiming miracles, snake oil and an app that will even cure the dandruff - just sign here. Of course, the clique has their cut - that's why they say yes or advice the board to say yes.

Then begins the grueling process of "analyzing workflows". Do they contact the actual users who are doing the work? Hell. No. What they do is, create a "Project Team" - usually hired anew, with no information about how the company does its work - and they try to "understand" the workflows. Then it becomes like that game, user says one thing, project team understands another and says a different thing and the outcome is a different product that solves a problem but not the user's problem.

Of course, this process burns money. You gotta do development, you gotta have a server to run the app, you have to book meeting rooms in hotels to train the users, you have to create fliers internally to promote the app - and create pdfs, many many pdfs to make the users understand how the app works. And no one asks "hey, if this app is reflecting our workflows... why are we getting this training?"

Because at the end of the day, this app only exists to make some people money. And after a certain point, no one dares to say anything because of all the money spent. An ambassador who says "the app we spent $10M does not work" will be get shot. People get retired with the f-you money they gained and the company tries to work with the app they "built" usually it ends up hiring an internal team and do it from the zero - and the expensive shit becomes a thing nobody talks about, a company omerta so to speak.

Another difficult scenario is when you are a lead on a project you know is bad. It’s a trap with no way out.

As a consultant I've been brought in on both good and bad projects. In one example, I figured out pretty soon the project was only kept alive because it was deemed high priority by upper management. I gave the engineering manager and project manager my recommendations for how mitigate a few of the highest risk problems, and maybe salvage something. After not hearing back from anyone for a month I realized that nobody was going to risk making waves for an effort they'd already decided was going to fail. Mostly they were positioning themselves to not be blamed at the post mortem.

Not your company not your problem. This article misses the point that your senior engineers often do not have the political power to push back on bad ideas at most med-large orgs.

In corporate structures failing groups will have high visibility resulting in promotions. The senior engs are letting those people get their money!

Letting it die is the self-serving, career-optimizing, amoral take. But it's more ethical to stand up for what's right even at personal cost. A bunch of people wasting years of their life, not to mention all the resources, is a tragedy worth avoiding.

Of course, the wisdom of taking the person risk is a continuum. In some cases it is and in some it isn't. But.. To omit the ethical angle entirely seems like a bad take.

Thanks for the insights

I've seen the opposite as well, and at a high level.

In this case it was an incompetent VP of Engineering who was seriously lacking domain knowledge when a new set of projects outside the norm came into the company. Instead of having a professional attitude, understanding his limitations and convening domain experts to help him and the team move forward, he actively opposed and derailed the project.

What's sad is that we, as external engineering consultants, were yelling at the top of our lungs trying to make management understand the serious liability this had revealed. They were absolutely blind to it until even a toddler could recognize the issue.

This cost the company millions of dollars as well as market reputation.

I think he is an Uber driver now, it's been a few years.

Every time I read this guy's blog posts, I'm very glad i don't have his career.