> just mess with it and see what happens

And even if you know every detail, that's still the best course of action, I think. Which kind of antialiasing you prefer, and how it trades with performance and resolution is highly subjective, and it can be "none".

There are 3 components to rescaling/rendering pixels: aliasing, sharpness and locality. Aliasing is, well, aliasing, sharpness is the opposite of blurriness, and locality is about these "ringing" artefacts you often see in highly compressed images and videos. You can't be perfect on all three. Disabling antialiasing gives you the sharpest image with no ringing artefacts, but you get these ugly staircase effects. Typical antialiasing trades this for blurriness, in fact, FXAA is literally a (selective) blur, that's why some people don't like it. More advanced algorithms can give you both antialiasing and sharpness, but you will get these ringing artefacts. The best of course is to increase resolution until none of these effects become noticeable, but you need the hardware.

The best algorithms attempt to find a good looking balance between all these factors and performance, but "good looking" is subjective, that's why your best bet is to try for yourself. Or just keep the defaults, as it is likely to be set to what the majority of the people prefer.