You just send packets to the other party's address and they send packets back to yours. Both parties know their address and you don't need a relay in the middle.