I'm going to give you a really strong counterexample to this. One of the most commonly acquired parasites that comes from gardening is toxoplasmosis.

But doesn't toxoplasmosis come from owning cats?

Nope. The correlation between cat ownership and toxoplasmosis is basically zero. Because toxoplasmosis comes from stray cats who poop in your garden and you catch them through the gardening. Something like 40% of toxoplasmosis cases come from gardening with people unaware that stray cats poop there. (The majority comes from eating undercooked food, like pork, because the pork grazed in fields where there was stray cat poop.)

So essentially doing your own gardening creates a risk vector that is significantly less present in industrial-level gardening of vegetables.