I have tried to forego swap because I had hoped that it would avoid swap thrashing, where performance goes into the toilet; I'd rather a process was killed but I could still access the system, than be unable to login to it.
But I've found that even with no/little swap, the system can get into high memory pressure situations (even easier because it can't swap out unused pages of libraries and the like), and still go into a sort of thrashing state even though it isn't swapping.
My main workstation is a Dell XPS15 with 32GB of RAM and I had 8GB of swap on it. About every month the system load would skyrocket even though it had plenty of mem+cache+swap available. If I killed off my browser and slack, load would return to normal, and I could use the system for a few more days or a week before it happened again.
I struggled with this for a year or more, before (for reason's I don't remember), I added another 8GB of swap. Within the next day 80% of the extra swap was used right up. I decided to swapoff that new file and create a 64GB swapfile and just see what happened. It got to around 20-30GB IIRC before it leveled off. And, more importantly, over the next 100+ days I never had that thrashing happen again.
That was with Ubuntu 22.04. I just recently (May-ish) switched over to a NixOS setup with 64GB of swapfile, and it's been very stable.
The old rule of thumb used to be 2x actual RAM for your swap file/partition size.
That's probably not right with today's RAM sizes but I'm not sure what the current rule is. I just go with the Ubuntu installer default but I don't know if that's just a fixed number or computed in some way from amount of RAM available.
Read only pages (most of libraries and binaries) get “paged out” to the file they came from, not swap.
(They obviously don’t page out, they are just discarded and read again when needed)