Whether they’re open source doesn’t matter (for this question). They control (their instance of) the server.
As you say, I do remember them issuing some threats about it, so it would be interesting to know if they’ve changed their stance on this.
(Discord, as an example, has banned users for using alternative clients.)
Alternative clients are banned in the Discord TOS. The Signal TOS is on their website and doesn’t prohibit any clients.
Also, separately, the idea that you can only use a service with a certain client is dumb.
Imagine if a website said you can only use a certain browser, or they ban you. It’s ridiculous.