It's mostly the audio clock that is suspectible to drift. Everything except the audio subsystem is derived from a single master clock, so even if the master clock varies in frequency slightly, all the non-audio components will remain in sync with each other.
That means the 2 clock cycles could theoretically make an observable difference if they cause the CPU to miss a frame deadline and cause the game to take a lag frame. But this is rather unlikely.
The CPU has shown some variation, but yes, it's the APU that has a ceramic clock source that isn't even close to the same among units. Apparently those ceramic resonators have a pretty high variation, even when new.
When byuu/near tried to find a middle-ground for the APU clock, the average turned out to be about 1025296 (32040.5 * 32). Some people have tested units recently and gotten an even higher average. They speculate that aging is causing the frequency to increase, but I don't really know if this is the case or if there really was that much of a discrepancy originally.
It does cause some significant compatibility issues, too, like with attraction mode desyncs and random freezes.