alt Hacker NewsWhy does SSH send 100 packets per keystroke?
Comments
Very interesting, I hadn't heard of this obfuscation before so it was well worth clicking.
Another good trick for debugging ssh's exact behavior is patching in "None" cipher support for your test environment. It's about the same work as trying to set up a proxy but lets you see the raw content of the packets like it was telnet.
For terminal games where security does not matter but performance and scale does, just offering telnet in the first place can also be worth consideration.
Wow, I did not realize that SSH did that. Good to know, and it makes sense as a default, because the people who need it need to have it on by default. But I think I'm going to be turning that off, because it's a security measure that doesn't make sense for my particular environment:
1) I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing. (Or rsync'ing a file, which ditto).
2) I'm not in a high-security environment where nation-states have an interest in sniffing my keystrokes.
3) I often open SSH connections to servers in other continents. Those underwater cables have massive bandwidth, but they're also in constant use by thousands upon thousands of people. So anything I can do to reduce my bandwidth by 100x is probably worth doing.
Any reason you can think of why I should not be setting ObscureKeystrokeTiming=no in my ~/.ssh/config?
Funny to see this fixed in 2023 and the side effects. Back in 2004, before I focused on performance, I did some security work including inter-keystroke latency analysis of captured SSH sessions to estimate the commands typed:
https://www.brendangregg.com/sshanalysis.html
The 2023 patch should finally fix that 2004 issue.
I don't see how Claude helped the debugging at all. It seemed like the author knew what to do and it was more telling Claude to think about that.
I've used Claude a bit and it never speaks to me like that either, "Holy Cow!" etc. It sounds more annoying than interacting with real people. Perhaps AIs are good at sensing personalities from input text and doesn't act this way with my terse prompts..
You can also use TCP_CORK to reduce the number of packets without any increased latency.
Disabling TCP_NODELAY would also reduce number of packets + be portable & simpler to implement - but would incur a latency penalty.
The reliance on LLMs is unfortunate. I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark. The Wireshark dissectors are quite mature, SSH is covered fairly well.
In 2023, ssh added keystroke timing obfuscation. The idea is that the speed at which you type different letters betrays some information about which letters you’re typing. So ssh sends lots of “chaff” packets along with your keystrokes to make it hard for an attacker to determine when you’re actually entering keys.
Now that's solving the problem the wrong way. If you really want that, send all typed characters at 50ms intervals, to bound the timing resolution.
Well, security is the #1 consideration for SSH, but if the author doesn't need security, why use ssh?
For example, "nc" (netcat) is pre-installed on all platforms where ssh is.
> That 20ms is a smoking gun - it lines up perfectly with the mysterious pattern we saw earlier!
Speaking of smoking guns, anybody else reckon Claude overuses that term a lot? Seems anytime I give it some debugging question, it'll claim some random thing like a version number or whatever, is a "smoking gun"
Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency. The big guys already do something similar with HTTP over TLS for DDoS protection and to limit end to end latency on TLS.
Granted... it would increase the cost (since you're adding reverse proxies) but it would be a quick way to get acceptable latency, rudimentary DDoS protection, and you could try different connection options independent of the main app's logic.
It would be hard to estimate how much latency you're adding with a SSH2 reverse proxy in this case, but it's probably lower than one might think.
The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.] But I loved this write-up. It was readable and explained the problem the OP was encountering and proposed solutions well.
"The smoking gun!" got me laughing, i am not a native english speaker and only ever seen that expression from Claude, and who knew? Its gaining popularity!
> I am working on a high-performance game that runs over ssh.
Found your problem.
But it is an interesting world where you can casually burrow into a crypto library and disable important security features more easily than selecting the right network layer solution.
Funny that this comes up today! I was just looking into adding a keyboard monitor to my website (I have a goal of making my 'contact me' page have oddly specific information). I wouldn't show the actual keys, just show a blinking light when there's activity, but I guess the timing really could expose quite a lot of information.
I did add a trackpad monitor though. It shows my raw MacBook trackpad data.
I enjoyed this write up as it touched on several topics I enjoy reading about.
Also I was unfamiliar with SSH being vulnerable in the past to keystroke timing!
Hmm, if the author is doing something high performance, they should probably use whatever mosh is doing to update the screen, not ssh.
The really mysterious part is how ~10,000 packets per second costs ~20% of a core. That would mean SSH is bottlenecking in its code at ~50,000 packets per second per core which would be ~500 Mbps per core (assuming full packets) which is ludicrously slow. It is trivial to do 10x that packet per second rate. Is SSH really that poorly designed?
Not related to SSH, but does the eieio.games website make anyone else's monitor flicker? When the website is fullscreen it overwhelms something. I thought my monitor's backlight was going.
I got 99 packets but an SSH aint one
I wonder if this is the same reason why Microsoft's Remote SSH plugin on VS Code is so flaky even with a decent internet connection. Every couple of months I try to give it another go and give up due to the poor keyboard latency I inevitably experience. And the slow reconnects whenever I glance away from my computer monitor briefly. This is on a fiber connection with a 20ms ping to the remote machine.
@eieio: whatever email protection you're running is triggering on the extension info. For example I see:
> And they’re sent to servers that advertise the availability of the [email protected] extension. What if we just…don’t advertise [email protected]?
Nice job! I need to learn how to use tcpdump apparently
fwiw, I tcdumped between two systems running fedora43 and saw no chaff. (one packet out, one reply, one tcp ack.)
> Keystroke obfuscation can be disabled client-side.
please never do that (in production)
if anyone half way serious tries they _will_ be able to break you encryption end find what you typed
this isn't a hypothetical niche case obfuscation mechanism, it's a people broke SSH then a fix was found case. I don't even know why you can disable it tbh.
I find it disturbing.
One thing you notice if you have ADSL is that some services are built as if slower connections matter and others are not. Like Google's voice and audio chat services work poorly but most of the others work well. Uploading images to Mastodon, Bluesky, Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for Tumblr you have to try it twice. I don't what they are doing wrong but they are doing something wrong and not finding out what they're doing wrong because they're not testing and they're not listening to users.
Nobody consulted me about their decision not to run fiber by my house. If some committee decides to make ssh bloated they are, together with the others, conspiring to steal my livelihood and I think it would be fair for me to sue them for the $50k it would take to run that fiber myself.
It's OK if you work for Google where there is limitless dark fiber but what about people in African countries?
It's the typical corporate attitude where latency never matters: Adobe thinks it is totally normal that it takes 1-5s for a keystroke to appear when you are typing into Dreamweaver.
>very confidently told me that my tcpdump output was normal ssh behavior:
I mean, for modern version of Openssh it's not exactly wrong. The failure was to tell you why that is the normal behavior.
Because we stopped coding for performance years ago.
>>> That makes a lot of sense for regular ssh sessions, where privacy is critical. But it’s a lot of overhead for an open-to-the-whole-internet game where latency is critical.
Switching to telnet instead of SSH might be an option.
> I am working on a high-performance game that runs over ssh.
Step one, run https://www.psc.edu/hpn-ssh-home/introduction/ instead Step two, tune TCP/IP stack Step... much later: write your own "crypto". (I'm using quotes because, before someone points out the obvious, packets-per-keystroke isn't, itself, a cryptographic algorithm, but because it's being done to protect connections from being decrypted/etc, mess with it at your own peril.)
If security doesn’t matter then why not use telnet or something else besides ssh instead of forking a security library?
I loled and closed the article after 'I am working on a high-performance game that runs over ssh.'
Vibe coders man...
[dead]
> I am working on a high-performance game that runs over ssh.
WAT. Please no.
> Obviously forking go’s crypto library is a little scary, and I’m gonna have to do some thinking about how to maintain my little patch in a safe way
This should really be upstreamed as an option on the ssh library. Its good to default to sending chaff in untrusted environments, but there are plenty of places where we might as well save the bandwidth