They are downsides in cases where an application has to escalate a critical fault as quickly as possible - indeed where every nanosecond counts - and a process waiting for page restoration is going to be delayed even an intolerable amount.
It should be a user switch and probably will become one in most Linux desktop distributations, I would wager. However, I would not enable it - I run Ubuntu Studio for every last little bit of latency it gives me in a realtime audio-capture application, and on those systems were it to become default-on by way of distro update, I'd be kinda miffed.