It's not "going after" the poor. You have to be out-right negligent to end up with any penalty at all, and the penalties involved are relatively small (20% of how much extra you owed them anyway). The vast majority of these low-income audits consist of "hey, we know you made money X, you didn't report it on your taxes. Fix it".
‘We know you’re trying to survive in your low income, but we want to extract more money from you so the billionaires don’t have to pay as much tax’
Cheating on tax credits also comes to mind