I fly back to the US pretty often (I am a US citizen living abroad) and have declined every time. This is in SFO. They are generally fine with it. But most people won't risk it.

It's much, much more annoying in Ireland, where US immigration happens in Dublin (an affront to Irish sovereignty, but that's another matter) - so being delayed can mean missing your flight.

> holding your passport.

When my spouse and I crossed through US customs this past spring, they called us by our names and waved us on before even getting our passports out to hand to the customs officer. This was at BWI, fwiw.

Customs or immigration?