> TLDR; folks should just use PostScript (Big) Points.

The distinction ends up being important if you need compatibility with some document format, or with common typesetting expectations. But if there weren't a concern of surprising people with certain expectations of font-picking widgets, I'd argue that the better choice would be millimeters.

4mm is a great default font size, and going up by one integer mm at a time is a reasonable step size (it's just under 3pt).