alt Hacker NewsNit. Surely it would be keyboard navigation? Your macro can consist of the key presses you would actually use, rather than trying to describe mouse movement.
If you want to automate from the perspective of what's on screen, tuis have no metadata. A button in a gui does have metadata associated with it, so you could automate 'click button x' where as in a TUI you'd be limited to clicking 'position x,y'
You don't really need metadata to automate in a terminal. Literally everything is there in the clear and ANSI is used for control. It's all characters; no need to think about pixels (I made something that automates TUIs[0] BTW.
For GUIs you have to handle the particular metadata of a given framework, if it exposes metadata at all for the controls. Otherwise you're dealing with raw pixels which is an instant nightmare, though not so much now there are visual LLMs, but you're still paying for the tokens and/or extra processing.
[0] https://gitlab.com/skeledrew/jaiver