I think this problem is avoided with swap on zram by setting swappiness high enough (>100). Theoretically, 200 would be best, although I'm reluctant to set it to an extreme value because I don't know how well tested that is. I was previously running swappiness=160. I increased it to 180 and I'll see if I notice any difference.