It goes beyond that. There is inherent classism in this, because it implies that you do not question the value of wealthy people who put in relatively little actual work output due to their privileged position. Take for example the unemployed person in your example who might have literally been 100 times more productive in their career solving substantive problems than a VC who lucked out on their startup and has been cruising on a few boards for ten years.