Aside from the practical and technical problems you're greatly limiting your audience. Most people who don't travel internationally don't have a passport. In Europe this might be a smaller number but in the USA and Canada I would guess this is a majority of people. Non-citizens won't have a passport. Most young adults will not have one. Many older people will have let theirs expire.
Yeah, that's the challenge. My bet is that there are enough people out there with passports to make an interesting social network.
Of course, getting someone to share their passport will be another filter. But I hope that I can convince people that the benefits are worth it (and that I will be able to keep their data safe, by only storing hashes of everything).