Swapfiles are fixed size just like swap partitions. They do not grow and shrink arbitrarily as you swap more/less.