Yes, I understand that part. The part I struggle with is how the very fact that a party without the key can do the computation on it is not an indication that the encryption is leaking information. If the encryption were airtight, then such computation shouldn't be possible.
Given that cryptography experts seem to be asserting otherwise, I assume that there's something important that I'm not understanding here.
The tl;dr is that breaking FHE would mean solving lattice problems that have been studied for decades to be nontrivial to break[0].
[0] https://arxiv.org/abs/2208.08125