Because rich and powerful people also have massive egos to go with their power and money. Some of them will find your rejection a personal attack and try to actively sabotage you. Not all VCs are like that, but there are enough of them that you need to be careful about how you handle the situation.