Infinite numbers break the abstraction of numbers in important ways.

Complex numbers do too (e.g. there is no ordering). But at least that doesn't cause as much confusion as some of the problems with infinite numbers.