It depends on incentives. From what I can see in CS, a lot of young researchers are focused on short-term projects, disconnected from actual problems, and spend lots of effort to package the result to increase the chance of getting a paper in top conferences/journals, because they need this for their career. They will be forced to leave academia if they don't have enough results in time, from what I know. And even for established researchers, they have to do something similar, so their students can have enough results. And they need to try really hard to get funding, because institutions want researchers that can get lots of funding, so institutions can get money from that.

This is probably not the complete story, and probably a bit too pessimistic, but I think this is true...

Up to certain point. Beyond that there is lot of waste.

Say you have 20 competing products that want to get noticed or know to sell. Obvious solution for them is to pour more and more resources in advertising. Eventually this advertising takes away from actual product as more time and money is spend on it instead on the product.

Advertising in academia is publications but also applications for funding.

No, only if you have a narrow definition of innovation. Producing something actually new and actionable isn't always aligned with doing something that gets citations.

Too much competition is bad for innovation since it leaves no room for exploration.

The peacock is a product of competition. Of course the competition was for passing on genes, not survival, so the peacock developed a massive tail which is a huge waste of resources and attracts predators. But surviving with such a handicap is super sexy to other peacocks.

Competition is great at meeting the criteria of the competition. If the competition values anything other than innovation, like say grant money awarded or social standing, it is suboptimal for promoting innovation.

As a former postdoc in the physical sciences (who is now out of academia mostly for family reasons), I don't like the constant argument I hear about whether competition is good or bad in research and especially academic research. I think it is the wrong question. Competition is inherently good in that whatever researchers are competing over will be optimized in the long run.

We wish that we were optimizing for new/great ideas, but we aren't. In our current academic system, we are optimizing for number of papers and number of quick citations on papers (where quick = within 2-5 years). The reason these incentives are present is because they are largely deterministic in the outcomes of academic hiring, tenure decisions, and funding proposals. It seems to me that everyone discusses academic hiring and tenure ad infinitum, but less so for the details of the academic research funding system.

For most academic research, when a professor submits a proposal for funding, it is tied very closely to work on one particular idea or group of ideas. The funding cannot be used for research outside of the proposal area. Furthermore one must achieve results within the confines and time period (a few years) of that grant if one hopes to receive more funding in the future. So when a new idea comes along during the process of working on a grant, you either a) do your best to spin the new idea as related to the current grant in some unnatural way and proceed or b) wait until you can get funding for the new idea explicitly. This is the system within which the professors must work. They are laser-focused on achieving results within the constraints of their existing grant proposals. And some of these are great research ideas. But after a while, most people tend to stick with the same old ideas and pursue smaller and smaller ideas within the same area. This is why old professors are still pursuing the same overdone research they did when they were younger. You need new, young people to give an influx of new/bold/crazy ideas to pursue.

Now, the graduate student or postdoc must also work within this system, except that they have no say over the research directions. They must work on the professor's research ideas, not their own. There's fundamentally nothing wrong with that because it is the classic master/apprentice relationship which is generally a good thing. (After all, you can't have well-formed ideas until you know what you're doing, and that takes time. Without this type of system, you get outlandish crackpot ideas that are worse than wrong - they are useless.) But over the years of training, the grad student/post doc probably has a few good ideas. But what do they do with those ideas? Generally...statistically...the answer is nothing. These good ideas die with the grad student/post doc's unrealized academic career, since by far most have to leave academia before they can work on their own ideas (and there's simply no place outside of academia to work on your own ideas).

You would hope that there would be an outlet for good new ideas from grad students and post docs, but there isn't. People learn from mistakes quickly that graduate school and postdoc is no time to be putting your ideas out there. You won't get to work on them yourself and they will be taken from you, period. Let's say you, as a grad student, propose something new and great to your professor, and ask if you can work on it. Chances are that the prof will say no because it isn't funded, or because you're already busy with their currently funded ideas that they must execute on quickly in order to get more funding, or the worst one (which I have seen many many times) is when the professor says "well that's more of this other postdoc's specialty - I'll let them work on it." Sometimes you could propose something and the prof says no, but then 5 years later they are now funded for it. And none if this is caused by malicious intentions: the professor probably forgot that idea even came from you - after all, how many conversations do you remember precisely from 5-10 years ago? - its just an idea that came from the ether somewhere. But other students and postdocs see these occurrences, even if not caused by maliciousness, and just choose to never share their best ideas because they know they won't get any attribution or recognition for them.

As a result, the system is not optimized for new and good ideas, which is the lifeblood of research. If anyone came along on this journey with me that I originally intended to be only a few sentences, I'm sorry I have no solutions. If anything, I feel lucky because 15 years later, at least someone else did one of my big ideas and it made an impact, so at least I get to know that "back in my day," I had some good impactful ideas in my research field.

It all depends on the metric you are optimizing for. In academia, the metric would be grant money, directly influenced by number of published articles and citations.

From that perspective, the system works. We are making more articles, in more journals, there is also plenty of money thrown around. Unfortunately, there is no incentive for correctness, novelty or usefulness in this system.

Falsification of results, especially in the soft sciences, is relatively easy. Verification of results, doesn't give you any credit. So you can have people producing articles with blatantly misleading or false results for decades, all without any repercussion.

And it's not much better in the hard sciences either. Because verification of results there, is even more difficult and costly. And again, we are not incentivizing verification.

It has to be a combination of competition and opportunity. If you take away the opportunity then it's just a middle school field day all day, nobody making it to play college ball let alone the pros.

I would guess that if there are economies of scale, concentrating production lowers the average unit cost and thus lead to more profits and thus more capacity for investment into R&D.

If, say, all steel production was done by 1-employee companies competing against each other, I don't think any one would be able to afford any serious investement.

Competition leads to innovation when the definition of success and its metrics are set correctly. A big chunk of "success" in academia is the number of publications, and funding often depends on that exact metric. As a result, competition in academia is very good for innovation in the field of producing papers and winning grants. I'm not saying the respective scientific research is wrong or doesn't exist, I'm just saying the system is skewed towards this one metric in an unhealthy way.

It's similar to competition in tech products not leading to innovation. The important metric there is financial growth and stock value, and there are ways to increase those metrics without really focusing on true innovation in the core domain.