Because society (random people you interact with) treats you differently depending on your physical characteristics. And dealing with a bunch of people who are mean to you for no real reason is harder psychologically to deal with than dealing with a bunch of people who are nice to you for no reason. Ask any person who was overweight and lost weight whether they noticed strangers being nicer to them after their weight loss.