I think you can reframe this and better understand the point these mathematicians are making.

The vast, vast majority of mathematics DOES use infinities. That's the standard perspective. The question is whether there is good, interesting, useful mathematics to be explored by disallowing that concept.

The way I see it, Gödel's, Turing's work and complexity theory come out of this line of thinking about _effective_ computation. This is an argument for exploring the mathematics that arises when you don't think of actual computer math as an imperfect approximation of the real numbers, but rather as a mathematical object in its own right.

I would guess (?) it's more interesting for floating point math and related than for integer math, because for integer math it's already well explored in group theory.