It's also a much smaller economy and Switzerland, unlike the UK, is landlocked and surrounded by EU countries. More than half of its exports go to the EU. Switzerland needs the EU more than the EU needs Switzerland.
European countries have traded with each other long before EU free movement. In fact, most countries today, EU or not, manage to trade just fine despite strict borders.
About half of UK trade is with the EU too and leaving made no difference.
As for "surrounded by EU countries", unless you're jumping in the same boat as joxdasba and claiming France, Germany, Italy and Austria will all simultaneously attempt to starve Switzerland into submission, that just doesn't matter much.