Did some debugging. It works after I disabled compression and control master.
I didn’t like CLI, since it didn’t pick up my default ssh key from the agent.