My issue with running most monitors this dim is that colors look so much worse and contrast is lost. It’s not as much of a problem on my OLED phone though, which does tend toward the dimmer side. My desktop monitors (IPS) on the other hand are rarely below 50%.
I have never noticed that problem. But I always adjust brightness as part of a full color calibration, with ICC profile and everything, which probably makes a difference.
I will say that it took a bit to adjust to the first time I did it, but now it's painful to use any monitor at "full" 200+ nit brightness.