alt Hacker NewsYou don't really need metadata to automate in a terminal. Literally everything is there in the clear and ANSI is used for control. It's all characters; no need to think about pixels (I made something that automates TUIs[0] BTW.
For GUIs you have to handle the particular metadata of a given framework, if it exposes metadata at all for the controls. Otherwise you're dealing with raw pixels which is an instant nightmare, though not so much now there are visual LLMs, but you're still paying for the tokens and/or extra processing.
Ah ok. I was thinking from the perspective of automating the input, for a human to view, rather than automatically getting data out.