You are correct, but you omitted one complication: Clients trust Google's and Apple's servers to faithfully exchange the participants' public keys.
Sending public keys through the notification system is an unnecessary complication.
Which clients?
Isn’t that what Contact Key Verification solves? Or do I misunderstand how that works?
... and hold participants' private keys truly private, which you cannot verify without a rooted phone.
Apps (such as Signal) that care about end-to-end encryption do their own key management. So, Apple / Google servers only ever see ciphertext, and don't have access to the key material that's used for the encryption.