I've been working with notcurses recently and it is a full TUI that handles mouse events just fine. Runs over slow SSH connections and everything. The nice part is that you can fully operate applications built on top of it with the keyboard if you so choose, the mouse is just a shortcut.

Sadly the project is not really in a usable state at the moment. The documentation is incomplete riddled with errors, the code has some pretty glaring bugs, and it's close to abandoned. It's a shame because you can do some really amazing stuff with it.