On a tiny screen like that, I suspect 800x600 is probably high enough DPI to fake the lines themselves well enough to the point where the pixels aren't discernable to the eye.

This alone still wouldn't remotely resemble a real vector display...

They would also need to accurately simulate the glow/bloom of the lines, and the phosphor decay rate over time that leads to effects like the "trail" behind the bullets in Asteroids. That is all extremely feasible. In a lot of ways, much easier than emulating a raster CRT display.

However, I have never seen a commercial emulation product do this with any competency.

Presumably because the number of people who would actually care is not large enough to affect the sales figures in any meaningful way.