I believe I got rid of most of the issues with some .ssh setting which simply uses the normal xterm I believe. I can't check right at this moment, but apparently, simply using xterm and not the xterm-xkitty or what it's called does not actually make much problems when ssh-ing into a machine - not in my case at least.

It was a setting in some ~/.ssh config file though I believe, not a kitty setting.