It's because each transform was "destructive" (like filters use to be by default). What link & vector layers do instead is store a transform matrix, so each transform just updates the matrix instead of actually re-rasterizing the layer each time.

We were hoping to expand that feature to all layer types for 3.2, but we ran out of time to properly test it for release. It'll like be finished for the next minor release.