The problem is the ‘fraud safety’. Your ‘safety’ is likely to be some else’s inconvenience (or vice-versa), and credit cards are all about ease-of-use. Fraud-related costs are also one of, if not the largest single contributor(s) to payment processing fees.
Other countries solved this by doing payments in ways that fraudsters can't simulate, such as a cryptographic handshake with a chip on the card.
Seems like payment processing would do well to be a public good instead of a for-profit venture to me.