Do you think the performance can be improved if the representation of the level is different?
I've seen AI struggle with ASCII, but when presented as other data structures, it performs better.
edit:
e.g. JSON with structured coordinates, graph based JSON, or a semantic representation with the coordinates
If it struggles with the representation, that makes it an even better test of the AI's thinking potential.
These models can “code,” but they can’t code yet. We’ll know that they can actually code once their performance on these tasks becomes invariant to input representation, because they can just whip up a script to convert representations.
In the limit case, to an actual general intelligence, representation is superfluous, because it can figure out how to convert freely.
To the extent that the current generation of AI isn't general, yeah, papering over some of its weaknesses may allow you to expose other parts of it, both strengths and other weaknesses.