That's too simple. First of all, Pixel (which GrapheneOS requires) is one of the few Android phones with a separate secure enclave. GrapheneOS also applies a lot of hardening that other vendors do not: https://grapheneos.org/features#exploit-protection

This does make a material difference, e.g.: https://x.com/MetroplexGOS/status/1982163802188575178

That said, if a state-level actor is up against you, then it's hard to defend yourself against that.