I finally gave up on using catch-all, and now only allow e-mail to addresses @mydomain.com that follow a certain pattern. If someone were to specifically target me and look closer at all the addresses I use, they would quickly figure out the pattern, but I see no reason why a spammer would do that.