1800s? We had actual open borders with the rest of the Americas until, like, the 1950s.

Farmers at the time were super-worried about the shift, since they already relied heavily on immigrant labor. Their concerns didn't manifest as major problems for them mostly because until very-recently enforcement was (pretty much intentionally) half-assed, such that the border remained de facto kinda open for immigrant farm labor (even, and especially, the illegal kind).

Now that situation's arguably not good for a bunch of reasons, but we've never had a strongly-enforced border, and in fact didn't regulate Western hemisphere immigration to any meaningful degree within living memory. Changing that to a highly-selective system with strong enforcement of immigration laws to keep out a large majority of prospective illegal immigrants would be a totally novel approach to US immigration. (Good or bad, either way, you can't really appeal to US history in its defense, and "without it the country will be destroyed by immigration!" demands an answer for why that didn't already happen, to remain a viable point)