That's a win for society if the money is instead invested into something productive!
I never invested in gold because it is not productive. I don't have any money, either (other than pocket money), because I've invested all of it.
Gold is usually invested in as a hedge against inflation. It's not really the gold that goes up and down in value, it's the dollar that goes down and up.
People choose to hold non-yield-bearing assets when they believe the returns offered by current investment opportunities are not sufficient to justify the risks.
It is the miracle of modern capital markets that enables almost anyone to quickly and easily invest their savings in productive assets, but of course capital markets aren’t perfect. The availability of “none of the above” options (like gold) that remove savings from the pool of active investment capital is the essential feedback loop that balances risk and return.
But it's a loss if it's forced into risky investments that aren't productive.