Isn't it mostly bad when the attribution falls to the person ("you're crazy" or "you idiot") and less so, even tolerable, when the attribution falls to the action ("that's a crazy thing to do" or "that's idiotic")?
Or is this concern about where the attribution lands "toxic PC" in your view?
There's a subtle difference between the use of term as you have indicated, but in this case it was a woman using the term 'crazy' in reference to something not calling someone 'crazy'.
It's a very clear cut example of Toxic PC culture and also a 'double example' of him being 'actually abusive' in the pursuit of ostensibly cultural empathy.
I do not think his belittling in that way is justifiable as 'locker room talk' - however the use of the term 'crazy' is colloquially acceptable language aka 'locker room talk'.