As a former postdoc in the physical sciences (who is now out of academia mostly for family reasons), I don't like the constant argument I hear about whether competition is good or bad in research and especially academic research. I think it is the wrong question. Competition is inherently good in that whatever researchers are competing over will be optimized in the long run.

We wish that we were optimizing for new/great ideas, but we aren't. In our current academic system, we are optimizing for number of papers and number of quick citations on papers (where quick = within 2-5 years). The reason these incentives are present is because they are largely deterministic in the outcomes of academic hiring, tenure decisions, and funding proposals. It seems to me that everyone discusses academic hiring and tenure ad infinitum, but less so for the details of the academic research funding system.

For most academic research, when a professor submits a proposal for funding, it is tied very closely to work on one particular idea or group of ideas. The funding cannot be used for research outside of the proposal area. Furthermore one must achieve results within the confines and time period (a few years) of that grant if one hopes to receive more funding in the future. So when a new idea comes along during the process of working on a grant, you either a) do your best to spin the new idea as related to the current grant in some unnatural way and proceed or b) wait until you can get funding for the new idea explicitly. This is the system within which the professors must work. They are laser-focused on achieving results within the constraints of their existing grant proposals. And some of these are great research ideas. But after a while, most people tend to stick with the same old ideas and pursue smaller and smaller ideas within the same area. This is why old professors are still pursuing the same overdone research they did when they were younger. You need new, young people to give an influx of new/bold/crazy ideas to pursue.

Now, the graduate student or postdoc must also work within this system, except that they have no say over the research directions. They must work on the professor's research ideas, not their own. There's fundamentally nothing wrong with that because it is the classic master/apprentice relationship which is generally a good thing. (After all, you can't have well-formed ideas until you know what you're doing, and that takes time. Without this type of system, you get outlandish crackpot ideas that are worse than wrong - they are useless.) But over the years of training, the grad student/post doc probably has a few good ideas. But what do they do with those ideas? Generally...statistically...the answer is nothing. These good ideas die with the grad student/post doc's unrealized academic career, since by far most have to leave academia before they can work on their own ideas (and there's simply no place outside of academia to work on your own ideas).

You would hope that there would be an outlet for good new ideas from grad students and post docs, but there isn't. People learn from mistakes quickly that graduate school and postdoc is no time to be putting your ideas out there. You won't get to work on them yourself and they will be taken from you, period. Let's say you, as a grad student, propose something new and great to your professor, and ask if you can work on it. Chances are that the prof will say no because it isn't funded, or because you're already busy with their currently funded ideas that they must execute on quickly in order to get more funding, or the worst one (which I have seen many many times) is when the professor says "well that's more of this other postdoc's specialty - I'll let them work on it." Sometimes you could propose something and the prof says no, but then 5 years later they are now funded for it. And none if this is caused by malicious intentions: the professor probably forgot that idea even came from you - after all, how many conversations do you remember precisely from 5-10 years ago? - its just an idea that came from the ether somewhere. But other students and postdocs see these occurrences, even if not caused by maliciousness, and just choose to never share their best ideas because they know they won't get any attribution or recognition for them.

As a result, the system is not optimized for new and good ideas, which is the lifeblood of research. If anyone came along on this journey with me that I originally intended to be only a few sentences, I'm sorry I have no solutions. If anything, I feel lucky because 15 years later, at least someone else did one of my big ideas and it made an impact, so at least I get to know that "back in my day," I had some good impactful ideas in my research field.