alt Hacker NewsOh my god, and the author even supplied a "proof"[0] visual diff harness... that it replicates the original game pixel for pixel.
Just the cherry on top of great demonstration of our collective new superpower: asking computers to do something we can describe how to do, but would (probably) never take the time to do ourselves.
[0] https://github.com/terrapapagalli1516/quake-srp/tree/main/or...
Replies
The visual diff harness was probably how they got rid of a lot of visual bugs, just tell the LLM to keep going until the pixels match exactly as the verification criteria
Using pixel data might be unusual, but anyone porting a game with a replay feature is going to realize it's an easy way to compare implementations.
Yeah cool.
How long before the same thing is done to like, banking back ends? Wallstreet proprietary software? Amazons logistics and distribution systems?
It seems like we might be weeks/days/hours before a situation where someone back engineers and spoofs a system so pivotal to modern human society that the plug needs to be pulled.
I suspect it's inspired by gbaeval. Both have the "oracle" and other similarities. https://gbaeval.com/
It won't be long until we can ask it to replicate iOS or MacOS.
Apple needs to up its game, fast.
Imagine the power of this magic superpower in the hands of business owners....
Adding to what you both said. I run pixel checks against live websites in a real browser, and I stopped doing full-page screenshots early on: scrollbars, lazy-loaded images and font timing shift pixels between runs like clockwork.
What works for me is diffing small stable regions, one component or one flow at a time, with a small per-pixel tolerance. I also keep a DOM-level assertion in front of the visual check, so when something fails I already know what changed structurally before I compare two images.
For the game port it's genuinely harder, the renderer decides the pixels, not the test. But the region idea transfers: lock the viewport and only diff what's stable.
You must be fun at parties
Actually Claude just did something similar for me, as I'm working on something else with Quake.
On its own it decided to do demo playbacks and take periodic snapshots, and compare them pixel by pixel if the PNGs differ.
I was also working on a web-based port but it saw the original Quake code was not modified so it compiled a native version on its own to use for this.
It had to apply a small patch to make the game completely deterministic, but it figured that out on its own by reading the code.
It then asked me to record a demo with various elements, say an explosion or being under water, and visually verified that the screenshots had those elements present.
So now I have a solid set of tests to verify against.
Opus 5.5 Medium. Used at most 1% of the weekly limit of my $20 plan.
tbh I thought this was a commonly used technique even prior to LLMs? I know I've been using it extensively myself, but I was inspired by Dolphin's extensive visual CI system.