Other users. Anonymity only works if you don't paint a unique identification on your mask.
That said, I would like all users to look degoogled.
(Not to object the degoogeling, but:) To create a Signal account, you have to validate a phone number. So Signal is currently largly about privacy, not anonymity.
FWIW I wouldn't know why to message someone if I didn't know who they were. The same goes for them not knowing who I am. What would we even talk about?
Ahh okay, that makes more sense! Sorry for the snark.
Signal isn't anonymous. The server has access to your IP addresses and the server can at will build social graphs about the users.
Where Signal's safety in numbers happens is in normalization of end-to-end encryption. Users who want to protect their message content are not standing out from the rest, unlike in e.g. Telegram where the vendor sees with whom you enable secret chats.