The problem with infinity is that it's a hack. It is basically the NULL pointer of mathematicians. An instance of a number that has a special meaning that breaks the abstraction of numbers.
If you want to do things with infinity, fine, but then do it properly and write things like lim x->inf (your expression with x here)
> An instance of a number that has a special meaning
Lots of numbers have special meanings. The ancients didn't think 1 was a number, and later lots of people didn't (and some still don't) think 0 was a number.
> An instance of a number that has a special meaning.
Not really. There are infinitely many infinities. Infinite numbers are not particularly more special than real numbers, complex numbers, matrices, functions/operators, etc.