As I understand it, some of the silver was siphoned off in trade with China (via the trans-Pacific route to Manila). China needed continuing silver imports because silver was not used there in the form of standardized coinage, but rather in ingots that were weighed and subdivided, with inevitable continuing loss.
I've not heard the inevitable loss as a culprit for Chinese demand. IIRC it had to do with a failed paper money system triggering inflation followed by a reversion to silver for exchange plus a growing population and market forces from the new supply from the new world.
https://en.wikipedia.org/wiki/Global_silver_trade_from_the_1...
It was very surprising to see zero mention of the Manila galleon[0] trade route in the linked article, even if technically the question was about gold rather than silver. The simple answer to the question of what happened to the money was that Spain spent it! The impacts on SE Asia were profound and are still being felt today.
[0] https://en.wikipedia.org/wiki/Manila_galleon