I'm writing this comment at night, on my phone with its absolute lowest brightness; it is still too bright.
Maybe, just maybe, people aren't "holding it wrong"
Unlike screens produced light. To make them the same brightness as ambient would make them unreadable in my current situation. Yet a black background is much more pleasant. Why? Because the light being produced is less. You're right that the contrast matters. Black background and white text gets us closer to ambient light level while providing contrast. You simply cannot do same with black text on white background. I mean go to a dark room, put down the phone and ask yourself what the natural background is. It's black. That's the ambient level