It's basically in-memory swap. If you have swap then you already have the added latency (but worse) and I/O usage instead of CPU. Like swap, the downsides only come into play under high memory pressure.