The Luddites were about protecting their privileged hereditary position against competition from women and other outsiders. Not unlike the privilege of contemporary software engineers like myself.
...against competition from women, who of course infamously had never been part of the textile industry? Who knew the Molyneux sisters were the Molyneux brothers all along?
A lot of labor unions were about racism and xenophobia too, but they did agitate for better working conditions nonetheless. For sure it's not black and white.