I don't think there is a solution -- Signal advertises itself as having a sort of security that isn't really possible with any commercially available device. You have to trust more people then just the person you're communicating with; if that's unacceptable then you need to give up a bunch of convenience and find another method of communicating.

Fortunately, the parties that you have to trust when you use signal haven't been malicious in any way, but that doesn't mean that they can't.