Because their institutions are completely corrupt. There are no rules to the game so the powerful take everything. It is a smaller pie but if I can have it all, why do I care?

Prosperity correlates with powerful institutions (not necessarily democratic). But ultimately you need a set of rules that even the state has to follow.

With Russia it's precisely because of their fundamentals. Too much depends on the people in charge, because Russia never developed effective institutions to constrain their leaders. When they have a good leader, they take a step forward, and when they have a bad leader, they take three steps backward. Bad leaders are always more effective, because it's easier to make things worse.

Medieval Russia was not too different from the rest of Europe, with all the little monarchies and merchant republics. The period of centralization that followed was also similar. But the accidents of history almost all went the same way. Monarchies defeating merchant republics and free cities. Muscovy being too good at conquering its neighbors, turning Russia into a huge centralized empire with little direct competition. The assassination of Alexander II before he could complete his reforms. The unholy mess of 1917–1922. The idealism of Gorbachev and Yeltsin that turned their reforms into a disaster.

Brain drain at an unimaginable scale, which is a self-reinforcing mechanism because once it starts, governments will usually impose restrictions to try to stop it, which makes everyone else want to leave too.

A lot of it is because transportation is quite difficult. Even today huge tracts in Russia are without any roads. On the opposite end of the spectrum is the US, where even before the railroad and the automobile, transportation was quite easy, with navigable rivers flowing through the best farmland in the world.

Another big piece is how easy it is to invade; how hard it is to defend. History records 50 invasions of Russia. For centuries Russian looked in envy at the very fertile land in what is now Southern Russia and Eastern Ukraine, but dared not farm it because it was swept regularly by quite fierce nomads on horses. That farmland became secure enough to farm only about 250 years ago.

On the opposite end of the spectrum is Great Britain, which has been successfully invaded, but not after Britain got serious (about 4 centuries ago?) about having a good navy. When the risk of the investment's being destroyed by an invading army is low, more investment happens. And a navy profits more than an army does from technological investment: 100 years before Britain's industrial revolution, London was investing heavily in copper mining and copper refining so that it could give all of its warships a copper bottom.

The US is even more secure than Great Britain because any great power or middle power contemplating an invasion of the US would need to cross either the Atlantic or the Pacific to start the attack. (Yes, the invader could try to get Mexico to agree to host the invading army, but the US would probably find out about that plan and respond by either blockading Mexico or invading it if it doesn't immediately abandon the plan.)

Policies that work great for secure countries like the US and Britain work terribly for countries like Russia, China, Iraq or Iran that must always worry about a land invasion. The basic strategy the US and the UK used in WWII had been worked out by Britain during the Napoleonic wars. Japan and Australia can follow the same basic strategy. Russia cannot because except for a few heady decades during the Soviet Union and maybe in the 1890s, the best way to increase Russian security was always to invest more in the army what with the country's being so hard to defend against a hostile army.