>There's no known reason for all the matter to be over here

The problem is there is no known reason for universe to exist. As such it could have been created in any configuration possible - including one where we observe matter, but antimatter is beyond event horizon.