The path to compromise that you describe exists, of course. But it's not the only way. For example, a zero-day in SGX combined with privileged access to AWS infrastructure could compromise Signal's SGX setup without any knowledge or collusion by Signal or Intel. As you note, it is not unreasonable for a project of Signal's stature to expect to face adversaries with such capabilities.