You absolutely do want to swap to disk. There are lots of non-file-backed cold pages that are pointless to keep in RAM forever. You could argue that these pages shouldn't exist, but they do, so why not shove them to a disk?
wasting SSD writes for one. Especially if something leaks and it starts trashing
I want instant access to my rarely used pages just as much as my frequently used pages. It's only pointless to keep in RAM if it's literally never used. If we knew how to identify those pages we could just avoid allocating them in the first place.