One reporter asked him about the rule that says MPs aren't allowed to cover their face while it is in session and he responded with "Oh, so now the rules matter?"
The thing that still confuses me about the whole thing is none of the other "real" parties even put a candidate up. If a joke candidate can get over a quarter over the vote then it seems like a real candidate could have had a chance, even though the city was chosen specifically because it was his strongest district in the last election.
For American voters it is kind of like if someone called Trump on his corruption and he decided to hold a national vote for President claiming that if he won it would prove that Americans don't care about corruption, but held the vote only in Corpus Christi, TX and the Democrats didn't even bother putting up a candidate.
It costs money to campaign, it's not a seat any other party had a chance to win and it sends a clear message from the other parties that the whole stunt was a waste of everyone's time.
How are you confused? Given your US example, the only smart play for the Dems would be exactly what happened here. Sure, you could field a candidate and maybe win, but probably not and then you've lent legitimacy to his stunt. Or you could just let Trump run against Bozo the clown, and while Trump would most definitely still win, he would be made to look ridiculous.
I think there's a few things going on that make it a bit different.
Firstly, Farage picked this constituency to run in because it's deeply favourable to him. This is quite common for leaders of parties — if you can't even get your top guy to win an election, how are you going to convince other people to do the same? In the previous election, he ended up with 46% of the vote, so assuming the same people vote for him again, you'd need almost all the other votes to go to a single alternative candidate to beat him — that's very unlikely. So there's not much value in running to beat him.
Secondly, Farage was always trying to sell this as a people vs the establishment election, so it wasn't really in the interests of any of the other parties to indulge him in that. Either it's a resounding victory against the establishment, or another example of the establishment using its power to beat up the little man. (Ignore the reality that Farage is fundamentally part of the establishment — the question is more how Farage would attempt to sell the result.) So even if you take the approach that the big national parties ought to be able to field a candidate in every election, even if they aren't going to win, there's not much value there either because the result will be used against you.
Thirdly, the stakes aren't so large as if this were a vote on the President. Farage is the leader of his party, yes, but right now it's a fairly minor protest party, with only 8/650 seats in the House of Commons. They take up a lot of column inches, but in terms of actual power, right now they're not that important. Gaining that one seat in the Commons isn't so big a deal to the day-to-day running of the country — Labour would continue to have a sizeable majority and control the Government, no matter who won it. So there's no question of "who should run the country" here, it's essentially just a performance to the media and the people.
Fourthly, beating Farage here stops the disciplinary processes from happening, and those are probably going to be more damaging to Farage overall than him losing this one election. As long as he's an MP, he will need to deal with the fact that he failed to declare this particular donation, which means it's kind of in the interests of the other parties to let him stay as MP and hopefully face the music. In practice, the punishment will probably not affect him very meaningfully, but the rigmarole of going through the process itself and having his bribe and associations with crypto billionaires repeated constantly in the media will probably be more impactful.
And fifthly, this is kind of just an optional extra election that Farage has called. If the investigation finds fault (likely) and recommends a long enough suspension (more uncertain), then the people of Clacton can choose to recall Farage anyway, at which point there will be a "real" by-election. I've no idea whether that will happen, whether the people in Clacton will really want that to happen, or whether the other parties will abstain again, but I think because this was an election triggered exclusively by Farage for his own benefit, there is this perception that the election never really counted any way.
Your last paragraph answers your second imo. The other parties want to highlight that the election was held with the outcome already practically decided. By not participating they are harming the "people vs. the establishment" narrative Farage is trying to spin