You are right. What most people don’t understand however is that “the border” extends for 100 miles
https://www.aclu.org/know-your-rights/border-zone
This effectively means border patrol can do whatever they want in areas that do not involve a border crossing and involve citizens just living their normal lives
Note also that that 100 miles _is a radius around your nearest international airport_ and need not be what you think of as a border, i.e. a line drawn between two land masses.
That concerns more than 220 million people in the USA, who can be arbitrarily stopped in the street and searched.
Be afraid...