It's not just a patdown. They take you to a phone booth that has a direct line to some portion of the FBI IIRC, and they ask you a bunch of questions to confirm your identity. At least this is what happened to me about ten years ago when I lost my wallet in a different state and needed to fly home.
... and the law in most states requires only that you give your name and possibly your DOB to the authorities upon detainment. So as a purely academic exercise, what can they even do if you refuse to answer beyond that? Obviously in practice they will fuck with you or just straight up violate the constitution, but theoretically I'm unsure how they can continue to seize you after that.