The way I set it up at my shop was that everything would build on your PR, and so by the time it merged, everything was already cached and no one should see a rebuild... at most a download.