In most situations the restricted lane (regardless of how you pick who gets to use it), does in fact benefit everyone else.

Under high congestion traffic throughput plummets. Restricted access to one or more lanes lets you keep them flowing at near the peak, increasing the overall throughput of the system by much more than one of the congested lanes.