The trouble is that if you want math to be standardized/able to be described in a sort of "objective" manner (what mathematicians call a proof), you'd like to start out with a set of axioms that are not themselves provable in the traditional sense but from which everything else can be proven. If you leave out infinity, it turns out that your set of axioms isn't really powerful enough to do anything, much less the kind of math that physicists require to describe the universe. If you keep it in, your set of axioms is SO powerful that you end up proving things that don't seem compatible with the physical world. There is no objective solution to this problem. The mathematical community chose the latter because, well, it helped us prove cooler and more sophisticated stuff. Some of that stuff is a beautiful way to describe the physical world (try googling something like representation theory), while other things make us question our intuition about it (i.e. string theory).