One of the things I didn't get to do at my last place before leaving was that I was putting together a proposal to beef up our CI infrastructure to be more cores per agent. I was trying to figure out the math to prove we would save money by doing so.

I had a batch process I became responsible for that ran on the same VM as an important production service and every once in a while it became a noisy neighbor. It ended up getting moved onto a much smaller cluster for the compute part of the operation after I did some pretty aggressive traffic shaping work to get the cost down by more than a factor of 10.

At one point I got the code into a shape where I thought I could just run it directly on the CI machine, and the lower CPU availability would be offset by the higher IPC overhead and cache misses. Problem was that it was still about 2.3x slower running on 1/3 as many cores. But those VMs idled for hours at a time while waiting for a 10 minute job to be started.

Near as I could figure, if we doubled the size of the instances the build agents ran on, I could shut down that cluster and just about offset the cost of the more expensive build agents exactly, and speed up quite a lot of builds in the process (many build steps do not parallelize but some do).

If one other team had a batch process they could just run directly on the CI machines, that would be a net reduction in operating costs. We had quite a few teams doing batch processing. CI servers have a lot of tricks you can use to queue up tasks rather than running them concurrently.