The constraint here is the e-ink refresh rate. That constraint does not exist on regular screens, which is why I'm surprised neither mobile reader apps or even desktop software takes purposeful advantage of it: fast scrolling back with fine, per-page control, keeping page in-place and zero perceptible latency rendering full page. With a finger drag or click&drag movement, this can easily be even faster than with paper books.

I know this works because I've seen it with some PDF readers on desktop in the past, when I lucked into a combination of lightweight document and right "fit page to display" scaling. And, of course, this works natively in browsers with PgUp/PgDown, but only on the few remaining websites that haven't been thoroughly shittified (no "en-" there on purpose) with JavaScript and excessive styling.

To each their own, it has never really bothered me honestly.

When you go back in a physical book, you also put a bookmark (either an actual bookmark or one of your finger) to be able to resume reading where you were, no?