It's precisely the opposite actually; If you're wealthy you can switch citizenships, hide your assets in tax havens, and afford property and other assets to store your wealth. For the debanked, or those living in unstable or authoritarian countries it gives them a more stable way to store and transact, especially with massive inflation.

Pretty much everyone I know who talks shit about bitcoin is wealthy and privileged.

Look back in history and there have been many cases of poor people being pushed out of their homes, stripped of their valuable goods and forced to relocate, in that case it would apply to those who were not wealthy.