It's a mental thing. The phone is the doorway to all the things, and the reader is the doorway to reading. When I'm being intentional about my phone usage (not always, but most of the time), I defer to devices that are focused.

It's hard to explain to someone who hasn't used an e-reader, but e-paper screens just look more natural, they also don't require a backlight so they can be easier on the eyes. Combined with a matt display, you can comfortably use them in direct sunlight without having to faff about with brightness or finding shade.

Also e-readers have way less distractions than a phone.

In addition to some of the other great answers, it is cheap enough to lose. The unexpected breakage/theft of a cellphone ranges anywhere from annoyance to world ending disaster. So, I have to be a lot more deliberate in protecting my valuable phone.

The tiny ereader contains no secrets or valuables, so I feel fine pulling it out in an area where I might otherwise be reluctant to use my phone.

A phone is a gateway to much more than just reading books, so your brain might prefer it.

Also notifications etc

It's easier on the eyes. LEDs and related technologies give me eyestrain after a while; eInk devices don't (and many, like the X4 model, have front-lighting, also easier on the eyes than backlit screens.

Personally I much prefer e-ink when I read for more than 10 minutes or so. The LED light from my phone just makes my eyes tired, while e-ink + "backlight" (guess it's really "kind of from the front light" in reality) just feels like reading normal paper.

The font rendering in the official firmware isn't that great though, especially compared to even a old phone screen. I had much better text rendering with 3rd party firmware.

eInk displays are much easier to see outdoors than phone displays.

How many books have you read on your phone in the last month? Or did you instead open twitter/hackernews/tiktok?