On the other hand, every single observation anyone has ever made was a finite value of something.

Even when things are potentially infinite, there seems to be a conspiracy of the universe to make it finite. The size of space may be infinite, but due to a finite speed of light and the finite age of the universe we can only ever observe a finite slice. When attempting to zoom in to an infinite degree onto supposedly point-like particles, quantum mechanics makes everything fuzzy and finite again. Black holes appear to have infinite curvature at the center, but it's unobservable due to the event horizon.

This principle seems so natural that we even apply it in advance: In quantum field theory, there are infinities in the equations when done naively, which we deem so unnatural that we apply funny tricks which make mathematicians cringe, but they miraculously work out so well that we get theory and experiments to agree to an astonishing degree.