This is going to depend on the holder's relationship with the data subject.

If you send a friend an email, they may add you to their address book and set a reminder for your birthday while they're at it. They don't need your permission to do that, so why would a device acting on their behalf?

On the other hand, if a customer of mine asks for the data I have on them, do I have to send them my Personal Knowledge Graph facts about them? That's less obvious, and I'd lean towards Apple's helpful database being a liability even if it's local.