That's a pretty weird distinction to make.
I remember back in the 80s writing a CGA text-mode game (they were quite in vogue at the time), and (as I assume most programmers did) I used the video memory directly as the source of truth about the current state of the level.
OP's distinction about video being a raster-based signal that you feed into a regular TV-like device, rather than being vector based or hard wired lights seems sensible. As to how that video signal is generated is kind of irrelevant.
I don't know why being vector based would disqualify this from being a video game?
https://www.arcade-museum.com/Videogame/star-wars
It's from 1983, which disqualifies it from being early, of course.
The manchester mark 1 had a teleprinter as its output, and used a Williams tube as ram (https://en.wikipedia.org/wiki/Williams_tube). If the image on wikipedia is accurate, the checkers game only "displayed" itself incidentally, on the Williams tube, rather than actually outputting to the teleprinter. In your game, it would be like writing the current level to internal ram, rather than to the actual video memory. The Williams tube isn't really a TV-like device. It stores data on a CRT, but that CRT isn't visible to the user in general operation, as the read plate covers the "screen". Again, "first video game" is up to a lot of interpretation.
Also, saying that vector based video makes it not a video game is a little strange, given how common vector graphics were in arcades (eg Asteroids, Tempest, Missile Command) and the Vectrex