On the Vectrex you could only draw lines between 256 x 256 grid points, so in theory 800 x 600 with antialiasing would be enough. But dunno if it would have the same contrast, OLED is as good as you can get I guess.
Not really. One of the advantages of vector displays is the fact that the drawn lines are razor sharp with zero aliasing. Another is the fact that the hardware has very fine control over the brightness, allowing for very bright or very dim lines to be drawn. The bright ones are brighter than could be replicated with raster CRT displays, and combined with slow-decay phosphors made for some beautiful "trail" effects. A pixelated display of any sort can only yield a rough approximation at best.
On a tiny screen like that, I suspect 800x600 is probably high enough DPI to fake the lines themselves well enough to the point where the pixels aren't discernable to the eye.
This alone still wouldn't remotely resemble a real vector display...
They would also need to accurately simulate the glow/bloom of the lines, and the phosphor decay rate over time that leads to effects like the "trail" behind the bullets in Asteroids. That is all extremely feasible. In a lot of ways, much easier than emulating a raster CRT display.
However, I have never seen a commercial emulation product do this with any competency.
Presumably because the number of people who would actually care is not large enough to affect the sales figures in any meaningful way.