There are ~600k H1B visa holders in the US. the tech sector alone has ~10M workers, and professional workers are ~9x that again. That boogeyman represents < 1% of the relevant workforce.
“White collar” work is the majority of US employment. It’s unclear to me if you’re proposing sacrificing white collar for blue collar jobs, but that’s not a trade our economy overall wants to make.
Relatedly, the unemployment rate for US factory workers is 2.9%. This is a very low unemployment number - 5% is generally considered “full employment,” and anything below that indicates a labor shortage. So your hypothetical factory worker should probably just go get another job.
I don't understand the nostalgia for manufacturing jobs. My mom worked in a factory putting pickles into glass bottles. It was not her dream job. I can still remember how she smelled after a shift. But it was the only employment she could find in that village.
Things got better when we moved after a few years and she shifted into a healthcare job. White collar if you will.