The mechanism is somewhat dubious imho - why would dirt selectively have beneficial bacteria that other environments don’t? I could come up with some just-so stories about how more “virulent” (haha) bacteria are the only ones to conquer our more sterile world but it’s just a plausible tale.
I do subscribe to it myself but not out of evidence. It’s just vibes. Built my daughter a little dirt box with grass and edible herbs so when she would muck around at five to six months she could do it in there on the balcony of our San Francisco apartment and she enjoyed destroying the plants so at the worst she got some fun out of it.
To keep it simple:
For one it's just competition. If you have regular contact to a diverse set of microorganisms chances are higher that they will keep each other populations in check. The dose makes the poison applies here too in some narrow sense.
Additionally (or even mainly) our immune system always on guard (me/not me) gets to play around with the obvious stuff and not be overwhelmed or be "bored" but constantly trained/tamed. We have evolved to have constant contact to a diverse set of microorganisms which modulate our immune system e.g. through toll-like receptors.
As an extreme case some people develop some natural immunity to the tetanus toxin through regular exposure to a extreme variety of microorganisms, obviously this isn't reliable nor a gamble remotely worth the risk.
[0]https://pubmed.ncbi.nlm.nih.gov/3980089/
[1]https://pubmed.ncbi.nlm.nih.gov/6827147/
Edit: don't know why OP gets downvoted it's a perfectly fine entrance question/bafflement to motivate some deeper digging into microbiology/immunology.
It’s not dirt, it’s soil. Soil is a living ecosystem.
The Simpsons explained this one a while back, funnily enough https://www.youtube.com/watch?v=aI0euMFAWF8
Ecosystems have niches for aggressive behavior against participants that reduce diversity. It's why you find higher concentrations of Lyme disease in new housing developments than you do in the deep woods: it's the activation of a defense mechanism (in the Lyme disease case it has to do with there being a higher proportion of competent hosts for the pathogen: mice, but in general evolution has myriad ways if implementing this pattern).
If it's just you and one species of bacteria, you're closer to a circumstance that would call for the activation of such behavior than if diversity is high.
So it's not that the bacteria are intrinsically good or bad, but that the different circumstances call for calm or aggressive behavior, and all your friends going missing cues aggressiveness in species who have evolved by filling that niche in the past.