Run without swap or with very little swap. I'm serious. Your modern server has enough memory, way more than 4MB, and you care about consistent latency don't you? Also, swap is wearing our your SSD.
You can't be serious having both 'your modern server has enough memory' and 'swap is wearing out your SSD' in the same sentence.
Even 0.3 DWPD drives have years before wearing out and in your modern server you really should have something with >= 1 DWPD.
It will happen even without swap. Code pages will get evicted and everything will slow down to a crawl. I have magic SysRq enabled so I can invoke OOM killer from keyboard if that happens.