Signal isn't anonymous. The server has access to your IP addresses and the server can at will build social graphs about the users.

Where Signal's safety in numbers happens is in normalization of end-to-end encryption. Users who want to protect their message content are not standing out from the rest, unlike in e.g. Telegram where the vendor sees with whom you enable secret chats.