It's not the population cap, but that bilateral agreements with the EU depend on the free movement of people. Without it, Switzerland could lose access to the EU market, which accounts for 51% of their exports. Switzerland cannot just choose whatever they like from the agreements and cancel what they don't. They will have to accept whatever in order to keep in the EU common market, or the country will suffer massive troubles. Switzerland needs the EU much more than the other way around.