I'd suggest to use ssh keys inside the secure enclave on macOS that require your fingerprint for every new session [0]. That way agents cannot initiate anything, and you will always be aware when a new ssh connection is being made.

[0]: https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb...