That's going to make it less economical, but it still doesn't even fix it. Even implausibly assuming the cards are perfectly secure so nobody could extract the shared private key from any one of them, somebody who wants to share their authorization could just plug their card into an internet-connected machine and have it sign for anyone else at will. If you give them the ability to sign you might as well give them the private key.

The basic problem is that there are people who will have the credential but want to thwart the operation of the system. If you can't unmask them then your system is thwarted. If you can, your system is an invasion of privacy that would have chilling effects because you're demanding for people to tie their most sensitive activities to their government ID.