saying infinity is a mathematical symbol we can observe is simplifying it way too much, all mathematical symbols are abstractions.
i can observe two apples. i cannot observe infinity apples.
Mathematical concepts don't have to have an obviously physical analogue. I mean, you'd find it difficult to observe minus two apples and certainly tricky to observe i apples.
To my mind, maths is like a "what if?" puzzle and whether or not infinity makes sense in the physical world, there's still fun to be had by considering the consequences of it.
That also means that it can be interesting to consider limited number systems which don't have any concept of infinity.
Some might say that 2 is as made up as infinity. Let me elaborate a little - your brain together with society made an abstraction "apple", and only by not distinguishing between these "sets" of atoms you can have numbers.