Even without location they can get a pretty good idea of your location from your IP address or any other signals. Their neighbor also might have allowed access to their phonebook or something like that to make the connection obvious.