I'm probably missing the right terminology. I would have thought the feedback loop creates jitter of its own. Phase locked VCOs are generally noisier than a crystal alone. But maybe I'm overthinking it and the loop bandwidth can be made very narrow with trimmed crystal and long integration times or whatever.
The contribution of the crystal to the overall tank circuit Q is vastly higher, by several orders of magnitude, than the contribution of the varactor diode.
So while making an oscillator steerable inevitably increases its phase noise slightly, the increase is either ignorable in practice or cancelled by the control loop itself.
Your own answer is right. For atomic clocks, the integration times of the feedback loop are typically much longer than a second.