> This is only about the Schengen, Switzerland is not a part of the EU

Not really, the bilateral are a package and the EU doesn't want CH to pick and chose.

If freedom of movement stops, a whole lot of thing also stop. It happened the last time SVP got something similar voted on (introduction of quota for foreign immigration), on a smaller scale (erasmus and horizon which are the higher ed and academic research collaboration, CH was a heavy recipient of the latter).