Oh boy... so I actually wrote a thesis in graduate school on conversational implicature, Paul Grice, and various other theories of implying things.

I would actually agree user dwheeler here.

Whether or not you agree with Gricean implicature theory (I do not), the point is that making a claim about a group that doesn't exist is absurd. Absurd statements do not convey meaning, and language is a tool for communication, thus it is generally an assumed axiom that statements will have meaning. Here, even when people make borderline nonsensical statements, we assume there is a metaphor or language game involved.

So, by making a statement about 'all my hats', if the number of hats you have is zero, then any predication is absurd and the statement is absurd, so given an axiom of not making absurd statements for natural language, you can assume there are at least two hats. Obviously there are no formal rules here, but the functionality of natural language demonstrates that these heuristics exist.

https://plato.stanford.edu/entries/implicature/