Is participation in your off time mandatory? If so, the problem is not hackathons. If not, why not just… not participate? The folks who do often enjoy working on the problems, networking, etc., so I’m not sure killing them all together is particularly fair to folks who do get a lot out of them.
The lack of participation is often one of those things that management will quietly hold against you.