You are 100% correct, but this is still mostly fine: without SGX, you need to completely trust Signal, since it could trivially modify the server-side code. But with SGX, you only need to trust that Signal and Intel won't both collude.
The most likely attacks on Signal involve trusted insiders or configuration errors, and SGX mostly prevents these, since to exploit it, you'd need to bribe insiders in both Signal and Intel, or find configuration errors in both of their software stacks.
Collusion is certainly still possible, but it's much harder to pull off, since it typically requires nation-state-level resources to exploit. Signal does actually have nation-state adversaries, but the vast majority of other software projects don't.
(I personally think that remote attestation is the single biggest risk to the free software movement, but I begrudgingly accept that Signal is a very good use case for it.)
The path to compromise that you describe exists, of course. But it's not the only way. For example, a zero-day in SGX combined with privileged access to AWS infrastructure could compromise Signal's SGX setup without any knowledge or collusion by Signal or Intel. As you note, it is not unreasonable for a project of Signal's stature to expect to face adversaries with such capabilities.
> The most likely attacks on Signal involve trusted insiders or configuration errors, and SGX mostly prevents these, since to exploit it, you'd need to bribe insiders in both Signal and Intel, or find configuration errors in both of their software stacks.
Since there have been multiple SGX key extraction vulnerabilities already, all you would have to do is compromise Signal and then use the key extracted from any of those devices, and "compromise Signal" is the same thing you would have to do if they weren't using SGX at all.