Yes. And that doesn't have to be an addict family member or a burglar, it can easily be e.g. a landlord who sees that you have more money, and charges you more money. There are a lot of people in a position to make you give up what you have, if they see that there's money to be squeezed.
This is even more true collectively: as an individual, maybe you can get out by being 50% more frugal than the rest. But everyone can't be 50% more frugal than the rest. If everyone tries, it's even more likely that larger society claws it back through higher rents, lower wages etc.
Everyone can't be 50% more frugal than everyone else, but it's possible for "everyone else" to be something other than 3-4% (the current US savings rate).
The money to sustain a high savings rate without a collapse in wages/demand ultimately comes from the government deficit, or deflation I suppose. This can be sustained even without productive real investment, if people just want savings the money will just sit around or repay debt without bidding anything up in the economy.
> landlord who sees that you have more money, and charges you more money.
You were getting a good deal then. Can happen with "affordable" housing I suppose.