> Which means I have to send my geolocation data to apps to confirm I can use my token?

No, you don't need to send it there.