It's worth noting that the way Signal's architecture is set up, Signal the organisation doesn't have access to users' phone numbers.
They technically have logs from when verification happens (as that goes through an SMS verification service) but that just documents that you have an account/when you registered. And it's unclear whether those records are available anymore since no warrants have been issued since they moved to the new username system.
And the actual profile and contact discovery infra is all designed to be actively hostile to snooping on identifiable information even with hardware access (requiring compromise of secure enclaves + multiple levels of obfuscation and cryptographic anti-extraction techniques on top).
Perhaps you're right that they couldn't be compelled by law to reveal it, then! However, I can still find people on Signal using their phone number, by design. If they can do that, surely there is sufficient information, and appropriate means, for US state-side signals intelligence to do so, too. I don't think Signal self-hosts their infrastructure, so it wouldn't be much of a challenge considering it's a priority target.
Now, whether FBI and friends would be determined to use PII obtained in this way to that end—is a point of contention, but why take the chance?
Better yet, don't expose your PII to third parties in the first place.