Because distant celestial objects accelerate a lot of matter away from them as parts of different processes. For example if there were a sizeable chunk of space were antimatter than at the edge of this space there would be a constant glow of annihilation. We don't see that anywhere (well except things like quasars doing this at small scales.

So, yea, it would have to be outside the observable universe in which that means we can do no empirical science on it therefore it not being real to us.