I have absolutely no doubt that this is possible to do, particularly if you assume that you already have all kinds of libraries available, and if you don't care at all about the terminal ecosystem in general.
And then you only need access to the mouse position in pixel granularity, and you basically have the foundation for a graphical environment. We can implement Qt and GTK for that new thingy. So there is finally a usable text editor available in a Unix terminal! Email clients that don't make you sad! You can finally navigate your files in a less lousy way!
And, of course, we can then also port these E libraries, so we can start their terminal app inside their terminal app inside their terminal app!
But: What is it for? Why not use your graphical environment in a direct way? The existence of terminal emulators is the proof for it being at least as strong (or stronger) as your terminal can ever get. Right? So what's the point of this indirection? I just don't get it...
Yes... Let's imagine I regularly look through my files. And these files aren't plain text (otherwise it would just be cat or mcedit) and aren't ODT files, kdenlive projects, Gimp files, ..., ..., but they are particularly png or jpeg or mpeg (or whatever the tycat thingy understands). And I want to do that via ssh. And I always have this E terminal in range. Then this is one valid option to do so imho. Still a very weird, freaky, odd one. But it would somehow make some sense to me...
We do also use our graphical environment. It's just that our terminal also happens to not be stuck in the 1970s and pretending it's running on a teletype. Decades ago someone could have made a very similar argument to the one you're making that we shouldn't have added colours to terminals because real dumb terminals are all green or amber screen.
It's at least partially about pushing the envelope, not accepting the status quo, and trying to improve things. Terminal emulators tend to have a fixed feature set and there's a bunch of things they can't do that would be nice to have.
I mentioned the kitty terminal emulator before. It's doing similar things. And it's quite popular with the kids. These enhancements to terminals are a good thing! I'm glad these people are experimenting with things even if they turn out to not be very useful (and many terminology improvements are great!)
Another great example of this type of thing is the tysend command, which lets you download files without starting a new ssh session: you're ssh'd into some remote machine and you want a file. You can switch to another terminal and scp, or (as long as the host you're logged into has tysend), you can just do 'tysend /path/to/file'. Terminology pops up a (very pretty) save dialog asking where you want to save the file, and then displays a (very pretty) progress bar while the transfer happens.
I think maybe you need to try terminology to understand the many, many ways it's superior to a more conventional terminal emulator. For me, terminology is definitely enlightenment's "killer app". You can try it just by installing it, btw - you don't need to be running enlightenment :)