If your brokerage account is large enough that most of your change-in-net-worth happens in unrealized capital gains, your tax rate is 0%. That's pathological at the best of times and in a "capital wins" scenario it's positively thermonuclear.
By that logic, you should also consider what happens if the brokerage account goes negative. Are you willing to do that?
Folks like to hate on capital gains tax. I think it's perceived as a rich-person deal. But homeowners, do they get taxed each year on the increase in value of their house? Almost the same thing, but not hated on the same way.