What does "inferring the UI content mean". You said you have some knowledge of how it pulls it off, is it using accessibility API and apple events behind the scenes? Those are the only two APIs I know of on mac that can do this (besides just grabbing the pixels of a window).

I've always been hoping that the rise of LLMs leads to a resurgence in apps being able to respond to apple events, since it seems like a perfect fit.