> technically if you have 3 or more sources that would be caught; NTP protocol was designed for that eventuality

Either go with one clock in your NTPd/Chrony configuration, or ≥4.

Yes, if you have 3 they can triangulate, but if one goes offline now you have 2 with no tie-breaker. If you have (at least) 4 servers, then one can go away and triangulation / sanity-checking can still occur with the 3 remaining.