Your system is still swapping all your file-backed pages (e.g. .so files) unless you're mlockall()-ing everything. If you pin non-file-backed pages in RAM by running without a swap file, the system has a harder time keeping your frequently-accessed file-backed pages in RAM and can't do as good a job as optimizing your usage patterns.
All you're doing by running without swap is pessimizing yourself.
I think this problem is avoided with swap on zram by setting swappiness high enough (>100). Theoretically, 200 would be best, although I'm reluctant to set it to an extreme value because I don't know how well tested that is. I was previously running swappiness=160. I increased it to 180 and I'll see if I notice any difference.