CD pickup detects changes in the reflected light due to the reflective pits. As long as the scratches are significantly bigger than pits they will create lower frequency attenuation to the reflected light which won't affect the high frequency signal coming off pits. You will get occasional errors when crossing into and out of a scratch but that's just a few samples, likely those won't even make it through through the speakers. I have not tried but I imagine a very fine sandpaper could create the scratches at high enough frequency to interfere with the pickup.

But the label side is indeed very fragile as you can easily damage the reflective pits, only covered by a layer of paint. It's as same as a simple mirror, where the thin layer of reflective metal is very well protected from the front but is only covered with paint in the back.