> I always found it odd that scaling down an image now and then scaling it back to its original size 2 seconds later with the same tool resulted in a loss of quality

I'm honestly baffled at your surprise... say, if you crop an image, and 2 seconds later you enlarge it to its original size; do you expect to get the inital image back? Or a uniform color padding around your crop?

Scaling is just cropping in the frequency domain. Behaviour should be the same.