Physical SIM cards are just as secure as the security enclave on the phone. In Norway few years ago banks even used that for secure authentication that worked on dumb phones with local mobile network providers pre-installing the required software on their SIM cards.
But then to save cost including the support cost banks stopped and instead started to require a non-rooted Android/iPhone.
Yup, it would be so much better to have it tied to simcard (though it might not help so much with anonymity)
But I think there are still cell operators without sim card