If the wikipedia image is accurate, its technically not "displaying" the board, its just in ram. The RAM just happens to be visible. But you get into a lot of technicalities when talking about the "first video game", so its up to interpretation. There was the "Cathode-ray tube amusement device" in 1947 that, by some interpretations could also be the "first video game" https://en.wikipedia.org/wiki/Cathode-ray_tube_amusement_dev...
I think it is at least safe to say that PONG isn't the first
I don't think it's necessary for video RAM to be separate from code RAM. The BBC Micro game "Revs" runs code from the video RAM and sets the palette to make it look like blue sky.
https://en.wikipedia.org/wiki/Revs_(video_game)
CRT Amusement Device is IMO disqualified for not using any form of computer.
That's a pretty weird distinction to make.
I remember back in the 80s writing a CGA text-mode game (they were quite in vogue at the time), and (as I assume most programmers did) I used the video memory directly as the source of truth about the current state of the level.
OP's distinction about video being a raster-based signal that you feed into a regular TV-like device, rather than being vector based or hard wired lights seems sensible. As to how that video signal is generated is kind of irrelevant.