This is not a perfect analogy: when I choose not to join a club, I have a reasonable expectation of that club not declaring me its member, and not fighting me for failing to follow the rules that the club established.
It's really not complicated. You can be prosecuted for crimes committed inside another state even if you were not physically present in that state. When that state has ratified the Rome Statute then you can be prosecuted by the ICC because that state lends its jurisdiction to the ICC.
The 'club' is 'humanity' in this case, we're all in it together and that's it.
If you go to someone else's house who is a member of that club and has agreed to follow its rules, it's not crazy to think you'd be held to them there.
Which is all the ICC can do to nations that aren't signatories to the Rome statute.