It takes one OPSEC slip up for someone to link a wallet address to you. So yes, your transactions being public doesn't matter as long as you are cognizant of that 24/7 365 days a year.
You can resolve this issue by repeatedly tumbling your money, using the same tumbling scheme as everyone else. This will reduce the value of your wallet slightly, to pay the mining fees, but it's… hm. That sounds equivalent to inflation or tax, except that the lost money doesn't go towards anything useful: it just goes towards buying ASICs and burning electricity.
You can resolve this issue by repeatedly tumbling your money, using the same tumbling scheme as everyone else. This will reduce the value of your wallet slightly, to pay the mining fees, but it's… hm. That sounds equivalent to inflation or tax, except that the lost money doesn't go towards anything useful: it just goes towards buying ASICs and burning electricity.