That's literally how iOS works today. If I go into Settings > Privacy & Security > Photos, I can give apps None, Limited Access, or Full Access. Same with Contacts, same with the clipboard (where the per-app choices are Ask, Deny, or Allow).

> It is not fair that Siri is the only one that can access these things now.

That would be true if it was, but it isn't.